I've been hacked... NOW what?
 

I've been trying to upgrade my copy of debian etch to prepare for an upgrade to lenny. The server installation is 1 year old and it hasn't received any updates installed until today.

In the course of trouble shooting issues with the upgrade, I noticed an error message in the boot logs that said:

And when I posted that message seeking an explanation, I was asked to do this:

/bin/ls -version

That resulted in the following message:

and the person advising me responded with

In addition to being in total shock, I'm lost here and not sure how to proceed. Frankly, since I'm not sure how (or when) this happened to begin with I'm not sure if I know enough to avoid letting it happen again.

Any advice here from someone with experience in these situations would be GREATLY appreciated!

Can I rely on tools like chkrootkit and rkhunter to help me determine what rootkit is involved and how long it has been here. Or am I trying to stand up in a quicksand pool?

Thanks!

Well, I will give you one hint:

I think this is a pretty good indication of what happened. If no security updates have ever been applied, clearly that is the first suspect. I don't know Debian well enough to determine the number and severity of the security updates you missed, but certainly missing any security updates is already getting off to a bad start.

As your adviser said, backup only the server's content. None of the programs, or even the settings. He mentioned that you could copy the configuration files if you "check them", but that would assume you know what to look for, and no offense, but it doesn't sound like you do; so I wouldn't risk it.

Copy everything to known clean media, run a few checks on it to be sure it is clean, and then wipe the server. Install your OS, and keep it updated. Of course, go through the security basics before the new server goes online. Use strong passwords, disable anything you aren't actively using, and make sure you are using sane permissions.

What role does this server play? WWW, DNS, DHCP? That would be nice to know, as there are service-specific tips to help secure your installation as well.

Finding 'ls' in your bootup messages is kind of odd itself but 'ls' has moved from the "fileutils" to the "coreutils" package in or around 2003 so that is cause for concern. If this is really a root compromise involving the SHV5 rootkit then you'd also find references to libsh.so, ttymon and ttyload binaries, and the latter two also in some of the SysV startup files.

I agree the server should be reformatted and have the OS reinstalled from scratch but before doing that it would be good to investigate, come up with evidence of a breach of security and create a timeline of events. This may take some time and given the state of (mis) management the server is in right now might not give you much more facts than starting from scratch. While you ponder your choices in the meanwhile save detailed process, open files, network connection and auth data off-site, then make the server unavailable by stopping public services, raising the firewall except for traffic from/to your management IP (or range), if the machine is remote you'll only need SSH.

Okay, MS3FGX. You're telling me I deserve to be beaten. I think I just HAVE been. But thanks for the reminder that even after 42 years in the IT biz I'm STILL not perfect.

By the way, I mis-spoke when I said the ls -v command was in a bootlog. That was wrong. I actually ran a debian script (dpkg-reconfigure) and captured the output to check it. I noticed the ls error in that file and questioned it. That's how the rootkit was discovered.

Frankly, I felt I WAS being attentive to security. But I admit this is my first time admin-ing a "www" server where I assumed full admin responsibilities (as opposed to admin-ing a strictly internal departmental server with NO web access or a dedicated www server that was supposedly being admined by a dedicated server hosting company.) At my last hosting company I had a server penetration within weeks of receiving my server. I reported the problem immediately. It took 2 years to convince their admins there WAS something wrong and get them to fix it! But I was forced to do the user account rebuilds myself.

I never forgot that.

That's why I decided a year ago to become the admin for my server. If I was paying them to admin and they weren't doing it, what was I paying them for? The way I saw it, if I failed, I had no one to blame but me and in the process I'd learn a lot. At age 59 I admit I'm still learning.

My rationale for not installing Debian updates was I didn't want to be running a constantly churning (unstable) server. So, I decided I didn't want to be constantly applying updates. In deciding that I didn't realize updates released by Debian would include security patches I SHOULD BE applying. At my last hosting company my server ran the same OS release for 2 years without them ever applying ANY security updates. I had NO idea a linux server could be so easily (and invisibly) compromised from outside. I've learned an invaluable lesson.

But ours is a profession where there are no excuses for ignorance. Yes, I screwed-up. I'll do my best to not make that mistake again. But I'm determined to try again anyway.

BTW, I did download and run rkhunter and chkrootkit. They agreed the server is infected with SHV5.

So, I'm forced to rebuild. Mea Culpa! What more can I say? There's no sense beating me. I'm doing a fine job of that myself.

As I said above, I was wrong when I said the ls -v command was in a bootlog. I had run a debian script (dpkg-reconfigure) and captured the output to check it. I noticed the ls error in the log and questioned it. That's how the rootkit was discovered.

I realize the server will need to be rebuilt; but before I start that, I'd like to try to figure out how long this compromise has been present and determine if there is any way at all to lobotomize it or keep it from doing any further harm. If it has been there for months (I suspect it may have been), then rather than take 25 client sites down for weeks while I rebuild, I might conclude it's best to leave the compromised system running and keep client sites running while I work on building a completely new server or building a completely new drive on THIS server.

I know I may be whistling in the dark here. But most of my clients are small businesses and their www server needs are not complex - a few html pages, a web form or two and some streaming music or videos. Only two of them involve significant databases and both of those sites are mine. But despite their size, my small business customers really do NEED their sites to be up and accessible on the web. So, my goal here would be to avoid putting them completely out of the web business while I tackle this rebuild.

Thanks a lot for your comments and any feedback you can offer.

The problem is, now your server is affected, your clients might pick up something nasty from your server. Note that I don't know the shv5 kit, but I'd be worried about my clients.... you're in a tricky position.

This is what google showed try this:

http://sourceforge.net/projects/rkhunter/

There is a sticky with security references on the main page of this subforum.

And I recently stumbled over 20 Linux Server Hardening Security Tips; it might be of help.

Yep, Chris. You're right. I am in a tricky position. Frankly I've spent the last few hours looking HARD at this rootkit and what others have to say about it. It wouldn't be easy, but I've almost convinced myself it could be lobotomized if not totally exorcised.

Thanks, Wim. This is a very useful set of tips. I've browsed through about 7 of them. Then I put it in my "read and study when I haven't been at the keyboard for 21 hours" pile!

Thanks again.

Thanks Nigel. I installed & ran rkhunter & chkrootkit this evening. They hinted at the existence of shv4 and confirmed the presence of shv5.

I then researched and read everything I could about the shv5 rootkit and found 2 or 3 others who claim to have successfully removed this rootkit without a server rebuild.

I then carefully studied it on my server and believe it probably can be manually defeated and removed. It has certain characteristics that make it "relatively easy" to identify, isolate and remove.

I also tried to determine the original date of the compromise but was unable to do that based on the dates of the files involved. They date back 2 or 3 years before my server even existed.

For the record, I did confirm direct replacement or impacts on ls, ps, top, dir, ifconfig, ttymon, ttyload, netstat, pstree, lsof, find, md5sum and other system components.

Now can you please explain what you're referring to when you mention SysV startup files? That one went right over my head. Are you referring to changes in the /etc/init.d/ startup files? If so, I'd say there's no question THAT could have happened.

From what I saw, I'd say it's likely both the openssh and openssl packages on my server have also been compromised by this rootkit. In fact, I'm convinced shv5 contains files designed to replace the server's public (and private?) ssh and ssl key files. (cue ominous music...)

With THAT damage assessment in my head, I'm going to go sleep on this.

Thanks for the feedback, unSpawn. You gave me food for thought.

Problem is: An attacker might have had access to a root shell on your server, that means he could have done anything.

And it is easier to just reinstall from fresh media
than trying to track down what might have been changed,
especially on a running server.

@mase
Easier yes, and the attacker will, with the same ease, get in again if you don't know how he/she got in.

And that's only the ones you know about. I don't think you have any alternative but a full wipe and reinstall. This time, after you do the install but before you open it for business, try running Tripwire (or commercial version here) to make a baseline record of what your server looks like (file names, sizes, etc.). Then you can take regular snapshots and compare the current data to the original data...any files that have been altered will be listed, and if you can't explain the discrepancy then you know there may be a hacker at work.