Think I've been had... how did they get in?

TruckStuff · 08-11-2005, 03:55 PM

Quote:

Originally posted by Capt_Caveman
What was the specific cron job run at that time?

Code:

# cat /var/log/cron | grep "Aug  9 04:0"
Aug  9 04:00:01 server crond[20462]: (me) CMD (/home/php/backup/db-backup)    // <-- MySQL backup script
Aug  9 04:00:01 server crond[20464]: (me) CMD (/home/php/tracking/track-relco.php)   // <-- Nightly maintenance script
Aug  9 04:00:01 server crond[20466]: (root) CMD (/pub/awstats-6.4/wwwroot/cgi-bin/awstats.pl -config=server.domain.com -update > /dev/null)   // <-- AwStats
Aug  9 04:01:01 server crond[20476]: (root) CMD (run-parts /etc/cron.hourly)
Aug  9 04:02:01 server crond[20478]: (root) CMD (run-parts /etc/cron.daily)
Aug  9 04:02:04 server anacron[20909]: Updated timestamp for job `cron.daily' to 2005-08-09

The frist three cron jobs have been running for >6 months each and have never caused a problem.

Quote:

Originally posted by Capt_Caveman
Assuming it's cron.daily or cron.hourly what are the various cron jobs that could be run at those times? (check /etc/cron.daily/ or /etc/cron.hourly/ for anything relevent)

Code:

# ls -al /etc/cron.daily /etc/cron.hourly
/etc/cron.daily:
total 92
drwxr-xr-x   2 root root 4096 Aug  8 11:33 .
drwxr-xr-x  57 root root 4096 Aug  9 04:02 ..
lrwxrwxrwx   1 root root   28 May  2 09:57 00-logwatch -> ../log.d/scripts/logwatch.pl
-rwxr-xr-x   1 root root  418 Nov 19  2004 00-makewhatis.cron
-rwxr-xr-x   1 root root  276 Sep 28  2004 0anacron
-rwxr-xr-x   1 root root  180 Oct 19  2004 logrotate
-rwxr-xr-x   1 root root 2133 Nov 23  2004 prelink
-rwxr-xr-x   1 root root  104 Nov  1  2004 rpm
-rwxr-xr-x   1 root root   82 Oct 20  2004 slocate.cron
-rwxr-xr-x   1 root root  286 Aug 13  2004 tmpwatch
-rwxr-xr-x   1 root root  136 Jul  5 16:22 yum.cron
-rwxr-xr-x   1 root root   38 Aug  8 11:33 yum-update-check

/etc/cron.hourly:
total 16
drwxr-xr-x   2 root root 4096 Sep 20  2004 .
drwxr-xr-x  57 root root 4096 Aug  9 04:02 ..

The only non-standard script on FC3 is yum-update-check. This is a script to email me a check for yum updates every night (but not do them).

I'll check out the strace thing and see what I find.

Matir · 08-11-2005, 05:18 PM

I have a sneaky suspicion yum-update-check may be related, because its modification/creation time is awfully close to the time this thread started.

Mind posting the contents of the file?

According to your first tripwire report, the tripwire database was last updated before the first run of the current version of that file (Aug 8, 13: something hours). That script would've most likely been run at 4am on Aug 9.

sundialsvcs · 08-11-2005, 05:25 PM

You really need to get some kind of bootable-CD, like those little "emergency disks" that Red Hat used to supply that would fit into your wallet. You need to establish "something you can still trust."

From this "trusted" environment, you then need to, say, re-download (and check the signature of) enough "trustworthy" packages from your distro vendor, and reinstall them, to further expand the system portions that you can "trust." You should particularly observe what kernel-packages are loaded or available to load. This will assist you in deciding whether or not you have, in fact, been hacked.

For what it's worth and from what I see, I remain un-persuaded. This tripwire report demands to be explained and you must not rest until you have done it, but this is a tripwire that suggests "a great, huge splat" instead of a targeted modification. It does not fit the modus operandi that I would intuitively expect to see here...

Matir · 08-11-2005, 05:33 PM

Might be modus operandi of someone who wanted to make it look like it wasn't the modus operandi of a cracker.

But I still think the script yum-update-check might be related.

TruckStuff · 08-12-2005, 08:22 AM

Quote:

Originally posted by Matir
I have a sneaky suspicion yum-update-check may be related, because its modification/creation time is awfully close to the time this thread started. Mind posting the contents of the file?

I suppose its possible depending on how yum checks for updates...

Code:

# cat /etc/cron.daily/yum-update-check
#!/bin/bash
/usr/bin/yum check-update

Matir · 08-12-2005, 10:30 AM

I guess I must be wrong then. That's a very simple script, of course. I guess it's just an odd coincidence. What is in yum.cron? (Just to eliminate the obvious suspects)

TruckStuff · 08-12-2005, 03:30 PM

Code:

# cat /etc/cron.daily/yum.cron
#!/bin/sh

if [ -f /var/lock/subsys/yum ]; then
        /usr/bin/yum -R 10 -e 0 -d 0 -y update yum
        /usr/bin/yum -R 120 -e 0 -d 0 -y update
fi

Code:

# ls -1 /var/lock/subsys/
acpid
atd
crond
haldaemon
iptables
kudzu
local
messagebus
mysql
network
qmail
smartd
sshd
syslog
xinetd

/var/lock/subsys/yum is intended to be created when the yum monitoring "daemon" is on and:

Code:

# /etc/init.d/yum status
Nightly yum update is disabled.
# chkconfig --list yum
yum             0:off   1:off   2:off   3:off   4:off   5:off   6:off

Matir · 08-12-2005, 03:35 PM

is there any chance /var/lock/subsys/yum somehow was accidentally created at some point?

TruckStuff · 08-12-2005, 04:26 PM

Anything is possible.

There is no process that I am aware of on this box that would have created that file. But even if we suppose that they did, there would be logs of the updates in /var/log/yum.log (which there aren't). Also, I've reviewed recent FC3 updates, and those wouldn't explain all the files mentioned above.

Matir · 08-12-2005, 05:33 PM

Ah. Well, there goes that theory.

TruckStuff · 08-15-2005, 08:57 AM

Well, I'm now convinced. On Thursday, I installed a known-good version of the ProCPS package, just to see what would happen. On Friday morning, at 4:02AM, all of the binaries provided by the procps package (ps, w, kill, etc, etc...) were modified (md5s, times, size, etc...).

I've downloaded Knoppix and will boot off of that as soon as I can get all of our apps off of this server. Perhaps that will reveal something as to wtf is going on.

But now I go back to the original question of this thread: How in the hell did these guys get in?? I've got strong passwords on everything. No unneccesary programs running. Current versions of all packages that are running. All the usual bells and whistles.

Matir · 08-15-2005, 09:16 AM

That's a good question. The best way to get more information about what they're doing is by checking to see HOW files were altered. For example, maybe comparing the current 'ps' vs a known good one. It might help us to learn what their intent is.

TruckStuff · 08-15-2005, 09:24 AM

I've examined the output of 'strings `which ps`' and didn't see anything unusual. The result was similar to a known-good version of PS. I did an strace as Capt_Caveman recommended on ps, but I' haven't done enough w/ traces to know what's good/not good. Right now, I really want to avoid recreating the exact same scenario that allowed this person to get in in the first place before I reinstall.

TruckStuff · 08-15-2005, 04:21 PM

So have I stumped everyone? If the source of a security breah is unable to be determined, what it the SOP for most folks? Reinstall the same configuration and hope for the best?

Capt_Caveman · 08-15-2005, 05:53 PM

Again, the time of the modification suggests that it's a scripted routine running at the same time. It might be informative to replace procps package with a good version again and then set the immutable attribute (chattr +i filename) on one of the modified files and see what happens. Hopefully if it's some benign rogue process, then it may generate an error. Though it's interesting that logger was modified too...

When you take it offline and reboot with a cdrom distro, not only take a look at md5s, but also look at the rest of the file system for any new files and verify that /etc/passwd looks normal. Looking over the list of modified files, I noticed a number of filesystem manipulation tools. So make sure to take a look at the disk itself for any new or suspicious looking partitions. Before shutting it down, make a tarball of /proc and look at /proc/modules.

Any luck with collecting data with Snort? How comfortable are you with disassembling binaries?