I help maintain a shared hosting web server running CentOS 6.5. About once per week I was noticing that the server CPU load was high and determined that a bitcoin mining script was being run. I blocked it a few times through the firewall, then changed /tmp to be mounted on a separate partition with noexec. That has worked well for the past 2 weeks, but it is probably only a matter of time before they figure out how to get around that, so I want a more permanent solution. I am guessing the base issue is that there is some sort of script that needs to be locked down on one of the virtual hosts to prevent unauthorized files from being copied there in the first place. How can I determine how the hackers are saving their files to /tmp or what script is allowing this? Or is there a better way or additional things to look at to prevent this?

It sounds to me you're having a serious security issue here, no "virtual host" on your webserver should be able to write anywhere outside virtual hosts root directory.

Questions to answer:
What's your setup, how are your virtual hosts configured?
What are the permissions on /tmp? Those files created, what are owner & permissions for them?
Do you have world writable files/directories on your system?
What user is your webserver running as?
What webserver are you running?
Are you running some kind of web hosting system like Plesk, cPanel, ISPconfig etc?

Check the logs to find more details of what's going on, look at /etc/var/log/messages and /etc/var/log/secure to start with.

All I can think of for now, enough to get you started!

Is there anyone else with admin access that you don't trust?

Thanks for the responses. No, there is not anybody else with admin access, just our staff.

Answers:
What's your setup, how are your virtual hosts configured?
Running Plesk (shared hosting) control panel 11.5 under CentOS 6.5. Virtual hosts are under /var/www/vhosts/domain.com. Other than that, settings for most are unique for each domain (152 domains).
What are the permissions on /tmp? Those files created, what are owner & permissions for them?
Permissions on /tmp are 777. It is mounted with with loop,rw,noexec,nosuid,nodev.
The files used for mining were rwxr-xr-x (755) and owned by user apache, which is why I think it is some sort of php script allowing these files to be created. I see over the weekend, they figured out a way around the noexec and I see a perl process with high CPU being run by user apache. The server load is 1.34, so I'm going to leave it running for now in case I can use the process for troubleshooting this security issue and hope that I can figure out how they are getting the files in /tmp for a more permanent block.
Do you have world writable files/directories on your system?
As far as I know, just /tmp and /var/tmp
What user is your webserver running as?
User apache
What webserver are you running?
Apache/2.2.15
Are you running some kind of web hosting system like Plesk, cPanel, ISPconfig etc?
Yes. Plesk 11.5.

Check the logs to find more details of what's going on, look at /etc/var/log/messages and /etc/var/log/secure to start with.
I have spent many hours checking the log files, but I haven't found much useful. But that may be because I'm not really sure what to look for. I did see some IP addresses trying passwords on email accounts, FTP accounts and trying to access some default files. I looked at and shortly before the timestamps of the /tmp created files and I didn't see anything obvious under any domain's httpd access files (or error, or xferlog, etc.). I also checked logs in /var/log, /usr/local/psa/var/log (Plesk location for some log files) and /var/log/httpd, around the time period the /tmp files were created but didn't see anything obvious. I think I need to know what to look for to see what script or program is writing to /tmp. Looking at the process (in /proc/[PID]) I see the cwd = /tmp, exe = /usr/bin/perl, cmdline = /usr/sbin/sshd -i.

Any ideas?
Thanks!

Should be 1777.


Please be specific: right now your version should be 11.5.30 Update #47 and not any lower.


As you found a process may have a name that doesn't match its purpose. Similarly any file may have a name that doesn't match its purpose. There's a couple of commands that may help: 'stat' (MAC times, ownership), 'file' (it may say "upload.png" but is a PHP shell or a tar ball) and 'strings -an' (read any files contents). There's a couple things you should do:
- If you don't know what to look for run your daemon and system logs through Logwatch. Makes transgressions easier to spot.
- Verify the integrity of your system. Just in case.
- Run '\ps axfo pid,ppid,sess,uid,cmd --sort=pid' and 'lsof -Pwln' and look at any other processes parent processes, ownership, working directories, open files, file names.
- Check all users and system crontabs, shell histories, login records.
- Check installed software, I mean whatever the web server provides like CMSes, boards, shopping carts, photo galleries, themes, plugins, addons, etc, etc, if they're stale, altered and if any directories contain files that shouldn't be there. Linux Malware Detect (LMD) may be of help.

And please respond timely.

Bingo ... "Plesk." The source of most insecurity in this world . . .

Also: "shared hosting." If you share a CPU with other subscribers who have nothing to do with you, then there's a lot of common property that all of you share, including perhaps the ability to drop into your "private" home-directory and read everything that's there. (Some shared-host providers, even very well-known names, are disgustingly poor when it comes to compartmenting their data. But hey, it's cheap.)

The "/tmp" directory is a shared resource that everyone shares.

Google 'bitcoin mining botnets on vps'
there have been several new articles on said topic this past month or so with descriptions in detail how the malware works from beginning(scanning) to end (mining on hacked box)
This info will point you to the malware files to be deleted/exposed.
After that, lock the system down and update update update!

Follow zeroday vulerabilities lists on securityfocus site or similar to keep an eye out for new vulnerabilities for software you use and how to fix.

Draios | Fishing for Hackers: Analysis of a Linux Server Attack is an excellent write-up by the author the sysdig tool.

Good stuff.

2 members found this post helpful.

Nice tool, thanks for sharing! Unfortunately one should install such things beforehand, it does require an LKM, and not "disturb evidence" on an already (suspect?) compromised machine IMHO...