MartinM,

The log files and suspect code are still under investigation. Based upon the log files, there is evidence that you were unequivocally being targeted for a brute force SSH password crack. This type of activity continued over several days with changing IP addresses. In the log file auth.log.2 there is the following entry on April the 18th:
The brute force guessing attempts continued past this point. This indicates that it was an automated script that was attempting to guess you account and succeed. Undoubtedly at this point you had been root level compromised.

There are still some suspicious entries, like the games/go directory and the plesk file ownership, such that a Plesk vulnerability can not be ruled out. We will have to continue to look into this. In the mean time, you have repeatedly expressed interest in getting this system back on line. In that regards, we are able to offer some suggestions to help prevent a recurrence of this situation. The recommendations are in response to the information discovered during this investigation. You will need to perform a complete wipe and reformat of the system. After you have re-installed the system, I would then recommend a wipe of the free space to ensure no hidden garbage remains. To wipe the free space do the following (for each physical and logical partition) Note I highly recommend you google for this process too and read to understand it as it is a little dangerous. Given that you will perform this on a clean install, the risk is limited to having to re-install again. There are also tips to monitor the process using the kill command passing a USR1 signal:
If you have any questions or need assistance setting them up, please don't hesitate to ask. The list is as follows:

1 - do not allow root SSH authentication. Period. It is unnecessary and not allowing this may have prevented this situation. Allow SSH only as a normal, non privileged user, and then use su - to assume root when, and only when, required.
2 - use key based authentication only (with password on the key and turn off password authentication in ssh). This too will prevent a password guessing compromise.
3 - use a strong password to elevate root. Even though you use keys, there are means to gain user level access to a system using known vulnerabilities and zero day exploits arise. Should this happen, this becomes your primary line of defense.
4 - at a minimum use .htaccess with a very strong password on plesk. Alternatively bind the application to listen only to localhost and use an SSH tunnel and access the page under http://localhost. Your htaccess should require at least as strong username and password. Even better would be to use a browser certificate (self signed by the server's openssl) to authenticate the page. If you make Plesk available outside of localhost, make sure it is on an https page and require SSL.
5 - install a HIDS to monitor the configuration files for any changes. Examples include Ossec and Samhain. This will alert you to unexpected activity and changes
6 - install an application like fail2ban with active response to greatly slow down script attacks. According to your logs, the attacker spent several hours, repeatedly guessing at your password. Had you used an application like this, it would have taken exponentially longer. In fact, there is a high likelihood that they would have given up long before they scored.
7- avoid using FTP use either SFTP or SCP instead for file access. FTP is another no-win. It is transferred in plain text and is unnecessary.

Additionally, you need to make it a habit of periodically reviewing your logs. Running them through a tool such as logwatch will help filter out the noise. Keep note of attempts to scan and password guess your system, though this won't work anymore, and if you find a repeat offender use a blacklist in Iptables to permanently ban them.

Your PHP and HTML files are likely ok. You should review them individually and if possible even compare them against previous versions. Make sure that you use proper permissions on these directories. For example, they should be owned by root and readable only by group and others.

Lastly, there is a cert recovery check list, much like the intruder detection check list. I have a link to it and will post it once I find it (or if someone else has it, please post it). It would be worth reviewing for additional tips.

2 members found this post helpful.

Thanks for the update Noway2. This might be the checklist you're talking about. Just out of curiosity, what is the reason behind zeroing out the drive instead of a regular reformat and reinstall? I've seen it recommended before, but just don't know how a compromise could survive reformatting and reinstalling.

2 members found this post helpful.

Thank you, thank you, thank you all who have been involved and who have invested their time and energy.

I have to say that probably 75% of the actions I have taken were way over my head, and 25% made sense to me. But even the 75% that was over my head, has made me more aware of what has been going on and I see the logic behind it, so I have definitely learned from this negative experience, that's for sure.

I have also experienced first hand now, that the often heard argument that one of the major advantages is the power of the community with Open Source and Linux is 100% true. Even a nitwit like me has found out that the people here are a shining example of that!

Now my next step would be to save the data from my clients, which would be the PHP/HTML contents of their sites and the contents of their databases.
I will of course check them against backups that I have from before this incident.
getting the correct content will not be any problem, but if I might ask one more advise?
What is the easiest way to backup the database-contents so I can re-use that once all the steps to make a proper re-install have been taken?
In my GUI-life I would have gone into Plesk and made a simple backup from within the controlpanel and upload it once the server is up and running again to put everything back. Wouldn't even have to worry about which client owns which file, that would have all been taken care of. But since I want to keep my server isolated until this is all sorted out, I'm a bit (to say the least...) lost.

Could you point me to a good How-To, which will help me to get it done?

And are there any more files I should preserve and get to you to satisfy any remaining curiosity? Please let me know, I'll gladly hand them over if this helps to prevent others to become a victim of these dirty tricks.

In regards to that checklist, I think you might be talking about this one, which should help me to configure the machine in a proper and secure way.

In addition to Noway2's instructions, I'd add:

- Mandatory, system-wide password changes. Looking at the scam file in /usr/games/go/, one of the first things it does is package up and email a bunch of system files, passwd included. If that was run, your passwords may be in evil hands. You don't want them getting back in the easy way.

Data files should probably be fine, as would a mysqldump file. I'd be cautious about PHP files. It might be best to work from fresh, trusted installs of Joomla, WordPress and others just to be sure. Either that or be sure to compare any transferred PHP files to a known good copy so nothing altered gets transferred.

Look into the mysqldump command. It creates a text file containing all the SQL statements to re-create and re-popluate databases on a new system. I'd just create a mysqldump file for each database and then run that file once you have the new MySQL install running.

Maybe one of the experts could chime in here. Probably the most useful thing would be an entire system image, but personally I don't know how to create one that is useful for forensics. Slimm609 was able to identify the rookit as shv5, which apparently is a known rootkit, and we've got those files. So I think we have a reasonable handle on what was happening on your machine. What might be a touch lacking is in how they got in in the first place. Certainly the successful root login via SSH is one vector. It is possible Plesk was another. Again, one of the experts can correct me if I'm wrong, but at this point I'm not sure that there is much to be gained by investigating Plesk since we know that root via ssh was open to them.

Oh, and one more thing. I'd like to say thank you to Martin for sticking with this. Very few people who come here for help actually do the hard work of figuring out what went wrong. By far, most of them just do a re-install almost right off the bat.

1 members found this post helpful.

@OlRoy, Yes that was the list. Thank you very much. The reason I suggested wiping in addition to a format is that I had read that a typical format only creates the file system structure and that it doesn't actually wipe the disk sectors. I am honestly not 100% certain of how true this is, but I do know that filling the drive even with zeros takes A LOT longer than a "full" format with bad block detection. I seriously doubt that even if anything were hidden on the disk that it would cause any problems, but I figured that this would be a safe precaution.

@MartinM, we are going to make a command line warrior out of you yet! Creating a database backup and restore in MySQL is extremely easy using the command line. MySQL contains a monitor program that you enter into as follows:
you will be prompted for a password for the mysql user. If the database and user are on a remote host, the user is <user@host> instead of <user>.

Once you are in the monitor, you can use standard SQL commands. For example, show databases; use <database>; describe <table> and forth. To create a backup of a database, you use the following command:

of course database-name is the name of your database and backupfile.sql is where it will be store. There is nothing special about these names.

Once you have the new database up and running, you can use:
Here are a couple of links with some information. Link one, two.

You should probably make a copy of your configuration files, which are mostly in /etc. If you modified any of these from default, it could be helpful to have them to compare if you find services don't work as required. Aside from that your design files would be most of what you need.

Here is a link to a good site (I think this is also available from the book review tab): http://ftacademy.org/materials
I would recommend the gnu Linux basic book for you. It contains a wealth of information on using the command line. Not only will you ultimately come to appreciate the power and flexibility of the command line, not having X and desktop managers installed increases security.

1 members found this post helpful.

Went to the attic, shoved some boxes around and found the Linux Bible by Christopher Negus. Gonna keep that on my desk the next few days

Will have to spent some time with the family now (it's about 10 in the evening again) and will start on "rescuing the data" tomorrow.

I will keep you guys posted on the outcome if you're interested.

When a file is deleted as you know it just deletes the link to the file. When you format, it deletes all the links to all the files. Files can be recovered after a format, but it requires something like The Sleuth Kit, foremost, scalpel, etc. To my knowledge there isn't a way for the deleted files to come back and reinfect the host. The only thing I can think of as being a problem is an MBR virus, which would be in the first 512 bytes. I've never heard of one for Linux, but that's not to say they don't exist.

Anyway, I'm no DBA, so I'll shut up now and go mow the lawn.

MartinM: also please be careful about the vuln.txt in the /usr/games/go directory. Please do not post the contents of that anywhere or give it to anyone else. We have the tar gz of it encrypted so you can delete it after everything is done.

1 members found this post helpful.

Note to self: Listen to the OP more closely. I reread the thread and noticed I missed MartinM said his first email was from Apr 18 21:12:50 warning about the possible compromise, which sure enough matches up nicely with the successful root login Noway2 found at Apr 18 02:01:14. MartinM was also the first to connect the sw-cp-server user to Plesk, which I also missed for a while. I missed some clues staring me right in the face.

Ok, as soon as it's over, I will get rid of it.

I have decided to use the CLI to make a Plesk backup, my train of thought: the more I get familiar with using a CLI instead of a GUI, the more I will use it.

Any files that have been altered after 4-17-2011 will be manually checked by me, using original source-files as reference.

I will do a complete re-install, make sure that SSH is not rooted (is that the proper way to say that?), use RSA, Iptables and other general standards taken from that CERT-document and apply the tips I have read here in this topic.

In regards to putting back specific settings in the server-config, coming from the current /etc/, the same method will be applied: Any file with changes after 4-17-2011 will be regarded as highly suspicious and treated accordingly.

Once all this has been done, I will restore from the Plesk-backup and go live again.

Since the reinstall will be done by remote hands, I think there is no need to "lift" the isolation from the server which is applied now, right? It's just a matter for them to put in a clean Lenny install and restart the machine, install and do the zero-fill thingy after that, correct?

Close but not quite. If you're going to do the zero-fill, you would do that before installing the clean Lenny. At least I would do it that way. And no, there is no reason to allow access to the compromised machine.

A couple of other things to think about:

- Noway2 mentioned installing Samhain or Ossec (AIDE would fall into this category as well). Those you would want to install as soon as possible. Since this is remote, you may want to put the same access restrictions on the new server as you currently have on the compromised on until you can get it ready and have your security installed.

- You might consider installing mod-security on Apache. It can be a bit of a pain to get the rules tuned, but given the number of PHP sites you host, it might not be a bad idea. At least on my server it does stop a lot of nonsense.

Thanks for the tips, I will be implementing them.

Though for me it would be easier if I also had the command to reverse the isolation

I know now how to isolate the machine, but I wouldn't know where to start to de-isolate it again

Are you looking to open up specific services or do you want to completely open up the machine again (something I wouldn't recommend)? In either case it would be useful to have the output of iptables -L -n so we know the starting state.

Sorry, was speaking double Dutch apparently

I meant that when I will have reinstalled, and I isolate the machine after that so I can apply all necessary security measures, I do know how to isolate it now, but after that I wouldn't know how to open it up again.

Ah, that is pretty easy. Something like this should work:

You can either do this as a script like I've got it, or you can do it as individual commands. If you do it as individual commands, be sure to set the table policies first (the iptables -P bit) because if they are set to DENY, flushing the rules(the iptables -F bit) locks you out of remote access to your machine. Someone with direct access would have to fix the problem.


Now, using these commands means that you have absolutely no firewall at all, which is probably not what you want. What you might want to do until you learn some of the basics of iptables is use a GUI tool like Firestarter (check the Debian repositories, I bet it is there). That will let you get a firewall in place sooner. In general, the idea behind a firewall is to deny everything and then allow only the things you want to allow. I've seen a lot of iptables firewalls that accept all traffic and then attempt to deny specific things, and that is not a good approach.


Believe me, your English is many thousands of times better than my Dutch. Heck, on most days, English is a second language for me, leaving me without a first.