My Server (Deb5 and Plesk10) is involved (causing) in brute force attacks
Linux - SecurityThis forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Lots of these lines: May 5 06:36:15 mmn001 kernel: [166954.174693] ssh-scan[16867]: segfault at 0 ip 8048e33 sp ffe223b0 error 4 in ssh-scan[8048000+c0000]
Now that is suspicious. Do you per chance have other segfault errors, especially and / or anything concerning PHP-CGI? I am guessing that this application is either the scanner or was the entry vector using the segfault to give them a root shell (hint - google the terms ssh scan segfault and you will get some interesting results).
I am not certain what to recommend about putting anything back on line. It is a risk. If you absolutely must, I would say do so by opening specific ports in the firewall, but you will need to concretely verify the integrity of the files in question as well as the server process that you would run. I would also caution to make an image of the system first, with the image you can always perform an analysis at your leisure.
Should find the scanner and thus the attacker's working directory, unless a type of rootkit has been installed.
You might also want to try:
ls -alh /usr/games/go
ls -alh /tmp/
The garbage isn't base64 and doesn't look like any kind of rot based on frequency of characters. Not that I'm a cryptanalyst or anything. I did some googling and apparently it has to do with the terminal. http://docs.intersystems.com/cache20...Y=GVTT_termdef
The other bit I'd add to Noway2's observation is that the screen session path is pointing to /usr/games/go (I see OlRoy noticed that one too). That might not be anything, but it strikes me as a bit on the odd side. Might want to have a look in there and see if anything looks suspicious.
Just a couple of comments on your log discoveries:
Apache error log - It's clear someone took/is taking a good hard look at your server for known vulnerabilities. Given the number of pre-packaged applications your serving (like Joomla and Wordpress) they may (repeat may, there is no evidence at this time) have found one. You did say that you patched everything, but was that post-crack? What was your patching routine prior to this problem?
dpkg.log - Did you do any of these installations? If not, we may have a date to start working with.
I'm with Noway2 on this, putting this machine back online is probably not the best idea. If you need to get your clients back online, I'd do it from a clean install.
Now that is suspicious. Do you per chance have other segfault errors, especially and / or anything concerning PHP-CGI? I am guessing that this application is either the scanner or was the entry vector using the segfault to give them a root shell (hint - google the terms ssh scan segfault and you will get some interesting results).
I am not certain what to recommend about putting anything back on line. It is a risk. If you absolutely must, I would say do so by opening specific ports in the firewall, but you will need to concretely verify the integrity of the files in question as well as the server process that you would run. I would also caution to make an image of the system first, with the image you can always perform an analysis at your leisure.
Well, I have over 5 MB of these segfault entries, it's in the mail on its' way to Hangdog42
Should find the scanner and thus the attacker's working directory, unless a type of rootkit has been installed.
You might also want to try:
ls -alh /usr/games/go
ls -alh /tmp/
The garbage isn't base64 and doesn't look like any kind of rot based on frequency of characters. Not that I'm a cryptanalyst or anything. I did some googling and apparently it has to do with the terminal. http://docs.intersystems.com/cache20...Y=GVTT_termdef
sw-cp-server itself is/could be legit, this has to do with the Plesk update-server, but the "backdor", hmmm....
Code:
mmn001:~# ls -alh /tmp/
total 88K
drwxrwxrwt 5 root root 12K May 6 19:09 .
drwxr-xr-x 22 root root 4.0K Jan 31 19:17 ..
drwxrwxrwt 2 root root 4.0K May 3 13:30 .ICE-unix
drwxrwxrwt 2 root root 4.0K May 3 13:30 .X11-unix
-rw------- 1 root root 41K May 6 16:51 autoinstaller3.log
drwx------ 2 root root 16K Jan 31 19:11 lost+found
-rw------- 1 root root 0 May 6 16:19 psa-installer.lock
-rw-rw---- 1 psaadm sw-cp-server 129 May 6 10:53 rkhunter.state
srw-rw-rw- 1 root root 0 May 3 13:32 spamd_full.sock
mmn001:~#
I think (duh....) that especially the second command (ls -alh /usr/games/go) is giving away some interesting info, to say the least.... (and certainly not limited to the two entries I marked red, the others are just as suspicious.
Last edited by MartinM; 05-06-2011 at 04:32 PM.
Reason: Additions
The other bit I'd add to Noway2's observation is that the screen session path is pointing to /usr/games/go (I see OlRoy noticed that one too). That might not be anything, but it strikes me as a bit on the odd side. Might want to have a look in there and see if anything looks suspicious.
Even I already noticed that one, since I am quite sure I have no user who is called games, at least not a paying one
Quote:
Originally Posted by Hangdog42
Just a couple of comments on your log discoveries:
Apache error log - It's clear someone took/is taking a good hard look at your server for known vulnerabilities. Given the number of pre-packaged applications your serving (like Joomla and Wordpress) they may (repeat may, there is no evidence at this time) have found one. You did say that you patched everything, but was that post-crack? What was your patching routine prior to this problem?
I'm always strict with that, when there are updates for WP or J!, I apply them as soon as I've seen the first reactions from others that their installations haven't been bricked. In reality this would mean that any installation from these CMSses would be up to date within 24 hrs after release.
Quote:
Originally Posted by Hangdog42
dpkg.log - Did you do any of these installations? If not, we may have a date to start working with.
I honestly wouldn't be able to tell you for sure, which imho means that the answer is probably "no". I just checked my mail to see when I received the first complaints from my hoster, and it turns out that this has been Apr 18 21:12:50, so this could very well be a start-date indeed.
Quote:
Originally Posted by Hangdog42
I'm with Noway2 on this, putting this machine back online is probably not the best idea. If you need to get your clients back online, I'd do it from a clean install.
I'm sticking to that too, I will go and get some sleep now (midnight in my timezone) and check back again tomorrow.
I want to stress that I am so grateful for all the great help I am receiving here, words are not enough (especially when you're not a native speaker, like me )
Already mentioned it in my previous post, but I'm gonna get some sleep (My 4 year old daughter will make sure I don't sleep in tomorrow ) and I'll be back tomorrow.
Everyone involved: Thanks, have a good one and talk to you later. I really appreciate your help!
Already mentioned it in my previous post, but I'm gonna get some sleep (My 4 year old daughter will make sure I don't sleep in tomorrow ) and I'll be back tomorrow.
Everyone involved: Thanks, have a good one and talk to you later. I really appreciate your help!
@MartinM Good Nite.
@anyone else Is there anything interesting in the logs that were sent to you around those ctimes on the April 28th and after?
I'm curious as to exactly what is in the malicious directory, specifically the "scam" and "secure" shell scripts, and of course the backdoor directory.
The three .tgz files have a creation date of april/may 2003, the 2 setup files are dated 03-12-10
And now I'm really gone
Looks like an old rootkit, and rootkits are one of unspawn's areas. If it was installed, it's not doing a very good job though.
When you get a chance, what are the last accessed times on those files? You should also try greping through your logs for events that happened around April 28th. Perhaps also using the find command to locate files that were changed on the April 28th. Like was mentioned eariler though, these commands could be affected by a rootkit. This attacker appears to be sloppy and with a little luck you may be able to important evidence in the logs. Here is a log cheat sheet for what you should be looking for.
BTW: Your root SSH password is a possible vector. Are you sure it was a strong one?
OlRoy, thank you for providing some top notch insight and some fresh eyes!
I think everyone might find this thread interesting. At least this isn't the first time this one has been around the block: http://ubuntuforums.org/showthread.php?t=1260606 It is an interesting thread in that it mentions the same files, with the same segfault violation. In that particular case, the primary suspect was a password crack by brute force, but the subject of an SSH buffer overflow with code injection was also raised.
I do think we have located the question code. Now, to find out how it happened. April 28th looks like a pretty good target date to focus on.
@MartinM, do you have the older logs, eg. auth.log.1 and the .gz archives? There may be some indications of a password attempt in older auth.log.
Also, what version of SSH are you running and how strong was your password? (the password you used should be permanently forfeit now too).
It would also be interesting to get a copy of this code. Please don't destroy it as some of us may wish to examine it.
MartinM: would you please tar up the /usr/games/go directory and mail them to hangdog if he does not have a problem with that.
Hangdog: would you please forward the other stuff he sent to the group also. If you don't have a problem receiving the other files also forward those as well. There may be some useful info in there to help track this down and maybe something useful for future detection and maybe some info useful for Rkhunter. If you do not feel comfortable receiving the files just say so and I will get in contact with him to forward and see what is going on.
@Noway2 No problem, thanks for kicking off the investigation.
I knew there was a segfault with ssh-scan, but not sshd. If that's the case, you're right in that an exploit against SSHd is a good possibility. Another possibility is like you originally mentioned in this thread, SSHd binary could of been altered, and that might account for the segfaults.
It will definitely help in getting things narrowed down if we know the Internet facing services and their versions at around the suspected time of the incident.
I'd also like to see the contents of /usr/games/ and /root/ because this "OLDPWD=/root" makes it seem like he was in that directory for a while. Also the last accessed time of wget, curl, ftp, or anything else that could of been used to download files. I'd prefer The Sleuth Kit to create a timeline, but I guess we've gotten this far without it...
With that said, the logs are our best source of evidence right now for determining the attack vector so I hope that rootkit setup script didn't include code to delete the logs when installing itself.
BTW: I noticed drwxr-xr-x 2 sw-cp-server 1000 4.0K May 6 23:15 backdor
Googled for that username and it's related to Plesk. Anyone have any theories?
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
