That is correct. Since the IP changing is a potential issue here is what I would recommend:

Block the traffic except for your IP. Once you have everything else cut off and only you are logged in, obtain a clean copy of the SSH binary. Configure this SSH server to listen on a non-standard port, configure it for RSA key only and only for your user account. Once you think this is working, open this one additional port in your firewall, but don't make it IP dependent. This will still let you in and will close off the attacker and any back doors. Then, close off port 22 to prevent them from returning.

I had already gone through the steps from your previous post and edited my last post before I read this one, there is additional info in my previous post.

Sadly enough it seems that my whole server is unreachable now, I cannot FTP to it, I cannot ping it and I cannot reach it through http.

Oops....

This happened after doing this:

I still have my console open and was able to run netstat, ps and lsof. But no http, ftp or ping. Is this what was supposed to happen?

Edit:

Next step was looking for setuid and setgid, which I did, results are in the attachment.

Attached Files

setuid and setgid.pdf (15.6 KB, 18 views)

Yes, that was supposed to happen. Everything except SSH should be cut off now (at least we hope so), and that is exactly what you want. You might want to test that you can establish a second SSH session from your computer just to make sure that you can access it.

2 members found this post helpful.

Just tried that and yes, I can

Also looked into /etc/inetd.conf by using "nano /etc/inetd.conf" which returned an empty file.
Another step from the CERT advise done.

Am hoping someone will be able to tell me something based on the files I uploaded here to see which process is the evildoer and needs to be killed.

Now it's time to get that new SSH binary and test that. Is that a matter of apt-get, or grep, or am I thinking in a wrong direction now?

Yes, that is exactly what is supposed to happen. The server has been locked down. Provided you used -s xxx where xxx is your IP, you, and only you, have access to it. The attacker has been cut off and the SSH attacks should have been stopped. The attacker can not get in any modify anything, they can't destroy evidence, they can't cover their tracks, and they can't command and control their applications. This is exactly what you want. Furthermore, things should not have been disturbed any more than was necessary and their applications should still be running so that you can identify them. If you need to upload any files to this system, use SCP, which works via SSH. The syntax is like this scp path/file user@yoursystem:/path/to/upload/to. If you have RSA key authentication working, it will work via the keys.

Now, I have looked over your list and there are some things that I think are potentially suspicious and some, like the Plesk related ones, I don't know enough to comment about. The files you have listed in /bin, /usr/bin, and /sbin are normal. The ones in /usr/lib may be normal. In my system (Slackware) I have a libexec directory with these files. The apache2/suexec can go either way. It is in a non-default place (per Apache) and I would put this on the to be checked list. The dmcrypt is one to check as it may be setuid, but if am reading things right should drop root privilege during execution (https://wiki.kubuntu.org/Security/Investigation/Setuid). I don't know what the /opt/psa stuff is but it looks like it might be associated with plesk. The qmail ones surprise me but it looks like there is a package for checkpw that uses setuid to avoid making patches. These files should be checked for legitimacy.

Now, there is a litany of things that need to be done and this will take some effort. The goal, of course, is to figure out how they gained entrance. In doing so, they should have made some noise somewhere. Based upon the command and process you found, I think that they gained root access.

1) You will want to get copies of all of the logs and examine them very thoroughly. Look for signs of intrusion. Look for signs of privilege escalation. Running the logs through Logwatch could be beneficial.
2) you will need to compare your system binaries against those for the distribution. Date and time modification may be of assistance. It depends on how careful the cracker was.
3) Look carefully at the process output - which I think help you find the rogue SSH program
4) examine all of the cron jobs
5) look for any hidden files
6) also, on your web server, are you running any sort of content manager, etc and if so what version(s). We will want to check the CVE list for known issues.
7) You said that Apache and everything were patched. That is good, but again check the versions against the security watch lists.

Remember, that any of these binary files, like find, could be designed to lie to you. You must verify them against the distribution first. If you need help examining any of the files and don't want to post them on line, I would recommend contacting any of the following individuals via private message (email): unSpawn, win32sux, noway2 (myself) unixfool, and Hangdog42.

1 members found this post helpful.

To verify your packages in debian, there is a tool called debsums. It should be the equivalent to rpm -V.

1 members found this post helpful.

Regarding the output of those commands. Please be sure to run them with the flags. For example, the output of ps shows bash and ps, which are the processes that your user was running.

The output of the netstat run (minus the process information) definitely shows an ssh bomb. Aside from the ssh connections, I noticed a few IMAPS connections. Are you running an imap server? You will want to try to cross-correlate the PID numbers and open files and sockets. Otherwise, it doesn't look like the offender was currently connected. Hopefully, this will lead you to a 'dud' binary, which in turn will hopefully lead to the entry path.

edit: I say that it doesn't look like he was connected as long as the imaps and imap2 connections are legit because the intrusion seemed to leave a screen session open with an ssh command. Speaking of this, you may see an entry for screen in your PS list. If so you may be able to switch to it with "screen -raAd" and disconnect from it with ctrl-A d (control A then d for detach).

The commands again are (run as root):

If I could ask a favor, could you re-run the netstat, ps and lsof commands Noway2 asked for? The ones you posted don't appear to have used the flags, and the extra provided by those flags provide is useful. One of the reasons I'm asking is that the ps output is ridiculously short, there is obviously much more than ps and bash running on the computer. If it remains this short, it suggests that at least ps has been replaced by something evil.

And yes, these commands do generate a boatload of data, so don't hesitate to contact one of us via email and we can get it somewhere visible.

<edit>

Noway2 and I are apparently thinking the same thing at the same time!

2 members found this post helpful.

I found that Cyberduck (on my Mac) does the trick, I'm a happy GUI-man again

Indeed all PSA-stuff is Plesk-related.

Now to be honest, I am more than willing to put in an all-nighter (and will probably have to ) and truly appreciate the help and patience I receive from you, but I need to know what to do exactly in layman-terms: Should I just download a copy of Lenny and check the filesizes and dates?

Or can you point me to a site where I can learn how to check these files for legitimacy and where to find this security watch list?

And about the logs, since I isolated the server I am not able to install Logwatch, so that's no option. Is there a trustworthy online tool for this somewhere around? And which are the most important ones to dive into, for as far as I can see, I have about 100 files in /var/log.

And in reply to hangdog42, another round of results :



lsof is still a pdf of 60 pages, so we need to find a workaround

Edit: Sent a message to hangdog42 in regards to that....

You are still not running the above commands with the options. What is after the command are called options such as "axfwwwe". Type exactly what was written in the code tags.

2 members found this post helpful.

Correct, I completely overlooked the options, my apologies for this mistake.

I have re-run the commands, with the options now.

Attached Files

	ps_options1.pdf (30.7 KB, 38 views)
	netstat_options1.pdf (20.0 KB, 22 views)

Let me start this next round by first complementing you on a difficult task that has been handled very well! In respect to, pulling an all nighter, you might want to re-think that (I imagine it is about 9-10pm already in your area). You will need to be fresh as you are looking for possibly subtle details. Let me also say this, if you are crunched for time in that you need to restore this machine to operation, make an image copy of it. I am not sure off hand how best to recommend this from a co-lo, but we will cross that bridge when we come to it.

Now for some specific instructions. In order to install any applications on this machine, such as log watch, you would need to download them from another location (get the .deb packages), upload them to this machine via SCP and install them with dpkg -i <package-name>.

You can find out what packages have been installed with the command dpkg --list, similarly, you can see what package a file belongs to with this command: dpkg --search filename

So, for example, from one of my systems running the Debian variant Ubuntu I did the following
1) which ssh
---> /usr/bin/ssh
2) sudo (or as root) dpkg --search /usr/bin/ssh
---> openssh-client
3) sudo dpkg --status openssh-client
---> This gives a lot of information. It tells me the package version, the dependencies, the configuration files (along with their sums) etc.
4) I installed debsums (you will need to do this manually)
4A) debsums openssh-client

The weakness here is that it verifies against stuff stored in your /var/lib/packages directory. This is a good first step and will likely show you if anything is changed. Running the init is probably futile at best and a bad idea at worst. you want to know if anything has been modified. I doubt if the attacker would go to the trouble of importing a whole set of cached packages. In order to be really sure, you will need to verify the md5sum against the package in the repository. You can browse the debian package repository against your version, find the files and extract them. You can then run an md5 against the downloaded copy and your copy. The debian packages are archives (like tar balls). See this link: http://www.g-loaded.eu/2008/01/28/ho...-deb-packages/ You can use these commands to extract the file of interest and then md5 it.

Note, there are a lot of files in a system. You don't have to do all of them, but I would start by focus on some key ones or ones that you think may be suspicious. You should probably verify your tools like apt, dpkg, and so forth. Use an md5sum utility that you have uploaded if you can. See if the find utility will give you any indications of things that may have changed in the last few weeks for example.

As far as the logs go, focus on /auth.log, dpkg.log, syslog, apache2/error.log, and aptitude

1 members found this post helpful.

Well, enough info in the logs, that's for sure.

Some examples that even I understand to be trouble

Apache2/error.log

Hundreds if not thousand of lines, all from different client-IPs but all looking for the same on my server:
[Sun May 01 06:30:53 2011] [error] [client xxx.xxx.xxx.xxx] script '/var/www/vhosts/default/htdocs/signup.php' not found or unable to stat

Hundreds if not thousand of lines, all from the same IP and all looking for different files in the same path on my server, e.g. phpmanager/mysqladmin/typo3/phpMyAdmin/etc:
[Sun May 01 05:37:54 2011] [error] [client xx.xxx.xx.xxx] File does not exist: /var/www/vhosts/default/htdocs/phpMyAdmin

syslog

Lots of these lines:
May 5 06:36:15 mmn001 kernel: [166954.174693] ssh-scan[16867]: segfault at 0 ip 8048e33 sp ffe223b0 error 4 in ssh-scan[8048000+c0000]

dpkg.log

Just entries from one day:

Auth.log

Filled with entries with usernames from same IP and failed passwords and entried with CRON sessions starting up and closing. Don't know if the last ones are correct, have to look in the schedule in Plesk to see if they are correct or illegal.

Aptitude

Empty log (should I rejoice or be worried? )

Personal Time-management:

True enough, I have to stay awake to be able to follow the tips and instructions in an orderly fashion. But on the other hand, some of the sites on my server are third-party sites and I feel a huge responsibility towards these people.
Is there any chance that I can make their sites available again without causing other server-owners on the internet to be flooded with those SSH-scans originating from my server again?

(I mentioned this to Hangdog42 off line and am adding it here)
In the ps output, I am suspicious of two entries on the second page: process 4268 and 4269. I think 4268 is the screen session that was left open by the intruder. I think 4269 is our real culprit. I admit that I don't understand how to fully interpret that entry with the command line full of garbage. I am wondering if it has been obfuscated like with bas64 or rot13 or something. What strikes me as odd though is the _/bin/bash entry, notice the _/ portion. I would look for a directory like _ esp in the root directory. Be sure to look carefully at the files pointed to by this entry.

The other thing I notice is a lot of entries of xinetd. These may or may not be ok. It strikes me as a little weird that it is listening on so many ports. I would look very closely at the configuration file associated with this application. According to the PS output, it appears to be tied to MYSQL, which I might expect, but it is listening on strange ports like 21 (FTP command) 106 (not standard), and 587 (smtp submission). You also have some things like the sw_cp_serv listening on public interfaces on high ports, which bears scrutiny.

The question remains, how did they get entry. Hopefully the logs will help answer this. I have a suspicion that they used a process vulnerability and elevated from there. The trick is to find which process.