Linux - SecurityThis forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
all packages on this server are up to date, including kernel.
First I would suggest to walk through the system maintenance step by step to ensure that the system is updated. There are some quirks regarding cPanel that this should be checked several times a year. Think of it as being about the same maintenance schedule as you would change the oil on your car.
I am not so certain that enabling SELinux would be the answer as it is still too experimental and there have even been recent exploits that target SELinux.
As the spam is originating from the root user, they may have exploited some of the system notification scripts.
Quote:
Originally Posted by pr1soner
two example exim's log lines:
2010-10-14 16:29:51 [24688] 1P6Pkd-0006QC-8o <= root@server.XXX.XXX U=root P=local S=635 T="Hi Joe Smith. It's Glenda. Wanna date?" from <root@server.XXX.XXX> for at18sharks@aim.com
2010-10-14 16:29:51 [24691] 1P6Pkd-0006QF-9B <= root@server.XXX.XXX U=root P=local S=659 T="Hello Pothead666. It's Charlene. Wanna date?" from <root@server.XXX.XXX> for bielfeedback666@yahoo.com
You would want to search against the Exim identifier to get more details on what all happened. Of course you may have to modify the command a little if the log files have already rotated.
Code:
egrep 1P6Pkd-0006QC /var/log/exim_mainlog
Click here to see the post LQ members have rated as the most helpful post in this thread.
// Apologies to the OP for this OT post but this needs to be corrected.
Quote:
Originally Posted by joec@home
I am not so certain that enabling SELinux would be the answer as it is still too experimental and there have even been recent exploits that target SELinux.
I'm currently having the same issue. Everything is patched, cpanel updated, exim updated. Same exact problem, same exact spam. What did you end up doing to fix this?
Last edited by unSpawn; 01-04-2011 at 11:02 AM.
Reason: //Merge back as this seems persistent.
Have you looked into the 'social engineering' aspect? When the software is or at least appears to be running normally the first thing I look into is who's had their hands on the hardware. While I'm considering that I'll check for any newly reported exploits and also plan a check for the possibility of a rootkit. (requires downtime, not always the most welcome request)
Exploiting these kinds of things isn't the hardest task, but it's by no means something a script-kiddie bot is going to accomplish, so long as you are up to date and firewalls are well established. It's not a bad starting point to consider either someone has targeted you directly from the outside (trade secrets?) or from the inside. (espionage for same, or disgruntled (ex)employee?) If you see no untoward external traffic making it's way in via logs, it's a pretty safe bet someone has messed it up from the inside, whether intentionally or not..
Only I have access to this server, it is a colocated server so support staff of the datacenter does not have access (I have IP kvm). It's an openvz node with about 40 vz's. Several of the VZ's are penetrated each night around 4-5 am. So far the VZs have little in common, some have exim, some just have sendmail, some have cPanel and some have directadmin. The node has all the latest updates and runs solusvm + openvz. The VZ's have all the latest updates.
My problem is identical to his. Same spam, same everything. I straced the process and it doesn't seem to do anything except spam through sendmail. I've checked for installed SSH keys, I've checked for changes to pam, checked /etc/passwd & /etc/shaddow. I also checked spool for any crontabs that have malicious data, nothing found . I have ossec running and monitoring those files for changes on the node and a couple of the VZs.
Hi,
Just wondering on this one, since you said it was a "data breach"
Can you be a bit more specific about what the file name was, and or contents of the script you found, etc?
I think it would be very helpful to others if you can provide some file specifics, as well as how you ended up solving the issue (that is formatting server, setting up a blocking command of some sort, etc.).
I read in another post on another forum that this was caused by an auto generating encrypted perl script found in / or /tmp, so curious what you learned.
Exim was patched on my servers. There are vps servers without exim that were also affected. Still no exact explanation for what happened, but os reload seems to have solved the problem. Perhaps it started with exim, but not sure how non-exim systems could have been affected. Exim was patched as I found out about the exploit some time before it was even public. What also concerns me is the fact that I couldn't ever find the method they were regaining access with (ie. the rootkit.)
Since the OP and you have consistently refused to share any useful information, since you have denied yourself any chance of investigation by "fixing" things and since I'm not into guessing any (further) reply that doesn't hold leads, useful information or clues is of no value at all.
Hi Dman1,
Were you able to learn anything more in your review?
I've experienced the same "Wanna Date?" hack and have to say it's quite maddening.
Even had all of the top cPanel support people review and they could not figure out how it was being generated from root (after over 4 hours of review on their part and another 20 hours on ours).
1. The spammer in this situation is somehow injecting the body of the email and list of addresses into the server root and sending them via perl as root.
2. Suffice it to say running every malware scanning script and file checking script known to man has turned up nothing out of the ordinary.
3. Even wrapping sendmail and logging to the nth degree did not help in identifying the source.
4. The hack itself injects up to 70k messages within the span of a few minutes, virtually the same time every day during a 2 hour window of time.
5. Other than the perl process appearing momentarily, watching the process list the microsecond messages are created and injected into the queue turns up no details.
6. Since we know when the spam is being injected we just freeze the queue for a couple hours, then the minute after the email is injected we just delete them all, so spammer is wasting their time and ours-- so this has turned in an academic issue at the moment.
- What we do know is that there is a script somewhere in root facilitating the sending of email through perl and sendmail.
- We know that some script on a site on server is being used to pass the information to the root level script.
- We know the spammer has a test script which emails an email alert to spammer approximately every 4 hours to check is server remains compromised (not a cron):
Subject: Notification # 6274
Body: All OK! ______ //h6vK3pJywC
- We know the script in root has a means of deleting itself, since we found a copy of the script on one occasion, and in later episodes look in the same location during the 5 minute spam injection and no scripts were found in that location or any other afterward.
So, if anyone has further evidence to share that would be helpful.
I'm basically stuck at the same point you are at. The perl script definitely runs as root, I can see it sending mail if I attach to it with strace. Standard penetration testing gets me nowhere fast. My problems persisted and I've taken a similar approach to yours. On affected systems I have a script running that kills their process before it can send anything out. Pretty crappy solution, but working for the time being. I was initially concerned that this was a result of exim exploits, but now know that's not true since qmail and sendmail servers were affected as well. I would guess this is a remote root exploit on a common linux service. I have not experienced the secondary notification script you have seen. By default I provision servers and vps that run exim to have a lot of logging enabled to deal with spam complaints and I don't see any mail like that being sent.
I'm basically stuck at the same point you are at. The perl script definitely runs as root, I can see it sending mail if I attach to it with strace. Standard penetration testing gets me nowhere fast. My problems persisted and I've taken a similar approach to yours. On affected systems I have a script running that kills their process before it can send anything out. Pretty crappy solution, but working for the time being. I was initially concerned that this was a result of exim exploits, but now know that's not true since qmail and sendmail servers were affected as well. I would guess this is a remote root exploit on a common linux service. I have not experienced the secondary notification script you have seen. By default I provision servers and vps that run exim to have a lot of logging enabled to deal with spam complaints and I don't see any mail like that being sent.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.