I've increased time between log rotation... that way if I'm busy or something comes on slow, I'll have plenty of data to study.
How do I send a copy of logs via email for safe handling elsewhere?

I'd never heard of a SAR until you mentioned it, so I'm schooling myself on that-- what I should have done in the first place.


I guess I never get tired of stating the obvious.

I have fail2ban, but it doesn't notice the most common exploit attempts I see in my logs-- php and 'myphpadmin' access attempts, so maybe I need to tweak it or add something else.

mod_security I'd heard briefly about but need to learn to implement... sounds like I have plenty to do.

Netfilter rate limiting is something I know nothing about, but I'm assuming it's limiting traffic through-put via certain protocols and/or ports?

Would it be worthwhile to set up an IDS (snort)?

Don't use email: let your stock syslog daemon handle that. See the basic example at http://www.rsyslog.com/doc/rsyslog_conf_examples.html and search the page for "remote".


Don't kick yourself for it. Just check what these tools can do for you, deploy the one you need and be happy they're provided free of cost.


What?


Either install mod_security, yes, and have fail2ban check its messages (http://www.fail2ban.org/wiki/index.php/Mod_Security) or enhance fail2bans http regexes (http://www.fail2ban.org/wiki/index.php/Apache).


Yes, the easiest example would be limiting ("-m limit --limit 30/s") the amount of new connections ("-m state --state NEW") your web server ("-p tcp --dport 80") should handle. The "fun" (not Spock's definition of fun) is you can combine several modules. For example you can limit connections and apply that rule to say any IP addresses in a /16 range, meaning limits won't "punish" or affect other ranges. The most important benefit of using the firewall is that you don't have to rely on any application level solutions: safer, more efficient.


That kind of depends. If your VM guest only allows TCP/80 traffic and only for mirroring then what it exposes will be very limited: no web-based management panels or interpreter-based applications remote hosts can muck with. This means its "attack surface" is limited to abuse of or flaws in the web server (configuration!) itself and the networking aspects of the Linux kernel. That in turn means the tuned rule set you deploy may be limited to filter for relevant risks like web server-centric flaws, any attempts to inject shell code and say known hostile addresses (all provided the guest is properly hardened first obviously). Try running Snort that way and then decide.

Ok, I believe I've set up rsyslog correctly.. will find out soon enough.

Pointing out that there are likely issues with primary care-givers in such people is stating the obvious.

Before I do that I need to see that I've got monit configured properly. After install, every time the server boots I get email notifications (and log entries stating) httpd failed to start, then monit tries to start it 5 times and disables monit monitoring of httpd. I'm not sure what I set up wrong there, because httpd is running all the while, but monit thinks it's not.


Ok, I have some things similar to this already in the mirror's firewall. I will probably post the firewall in a separate thread to see if anyone sees any potential problems in it since it's the first one I've put together-- and that with internet sources.


Ok, will do, and may look into setting up a honeypot (like honeypotd) to make that more worthwhile, at some point.

In the mean time I'm still waiting to see if it fails again-- hasn't yet since updating VirtualBox manually.

The ssh log (/var/log/secure) for the host looks like the following (logged every minute):
What sort of problem might this indicate?
**EDIT:**
*ding* <-- the light-bulb went on. This indicates Monit querying sshd once every minute. My initial monit setup was wrong for httpd, but apparently it worked (a little too) well for sshd.

I looked at my host's VirtualBox logs and also searched for information on the bug mentioned (re: ext 4, host i/o cache not enabled.

Excerpt of host's Virtualbox log:
I did as suggested and the warning disappeared. Also read the following on the Oracle forums (https://forums.virtualbox.org/viewto...?f=7&t=41683):
My kernel is 2.6.32, however.

I installed monit via yum (rpmforge), but that didn't work-- so I removed the yum package and reinstalled manually per http://blog.hostonnet.com/installing...-centos-server.

Apparently monit has a lot of ways it can monitor/restart:

This morning I found the vm in an aborted state again, so the VirtualBox update was not the problem. I set logging for the host and the vm to high a few days ago.

Here is an excerpt from the host's log sent out about 3 o'clock this morning:

Which other logs should I post?

Please let me know if I should instead start a new thread on this...

My VM will no longer boot ('kernel panic'), and this suggests a bad (virtual?) hard drive. I cloned the VM a while back and so was able to simply boot the clone in place of the mucked up original.

The warning I used to get when booting VirtualBox about risk of corruption-- I'm wondering if that's the cause of this, or perhaps just a coincidence. Either way, I enabled host I/O caching as suggested and was unable to find any further information about the bug or any other clues in the logs, other than a recommendation to update the kernel.

My apache mirror's kernel is currently 2.6.32 and the recommendation to fix the VirtualBox bug issue is to update to at least 2.6.36. My concern, having never done a manual kernel update on a live server before, is what will this do in terms of causing other problems? Sure it's just a volunteer gig and I can always go back to a clone if I mess this one up, but I'd rather not have to spend a ton of free (literal and figurative) time fixing other issues that crop up because of a kernel update that's not an official part of RHEL/CentOS.

The same concerns arise with going from CentOS 6.3 to 6.4. My question is-- assuming this might fix any problems pertaining to virtual disk corruption and/or a VirtualBox bug and/or any future kernel panics-- should I upgrade first to 6.4 and then the kernel, or vice versa, or does it not matter at all how I go about this in terms of prevention of headaches? Is there a smart way to upgrade the kernel that will mean less likelihood that anything else breaks in the process of updates not officially sanctioned by Red Hat/CentOS community?

(If the kernel update does not fix the VirtualBox bug, what then? My understanding is that KVM is still pretty unreliable for my purposes. Should I then switch to a free version of VMWare?)

Thanks much in advance.

Update:

It was a bad hard drive, I suspect. Will confirm with a little more time observing the server's behavior.

Basically, the stability got worse and worse, even getting to a point where the host OS was no longer accessible remotely, then I saw the following (rough notes follow) when trying to reinstall fresh OS (figured it was worth a try):

/tmp/program.log:
ext2fs_mkdir: Attempt to read block from filesystem resulted in short read while creating root dir

SCSI Mod ePage Offset:
response length too short
resp_len=47
offset=50
bd_len=46

>> terminate command early due to bad response to IEC mode page
log sense failed, IE page [scsi response fails sanity test]




So with that, I replaced the hard drive (luckily had a 2-day-old backup of my vm's on an old external hdd,) and since then it's been pretty smooth sailing.

Still trying to get monit to work properly though... on vm, it freaks out every time the vm boots, saying httpd isn't running and refuses to start, even though httpd is running fine. Need to fine-tune some things, clearly.

Learning a lot while studying for LPIC-1! Any suggestions for how to change careers and get into IT would be greatly appreciated! Hoping to obtain an MSCIS, but not guaranteed this option for some time. Certs? (CCNA, LPIC and RHCSA?)

No issues whatsoever since replacement of hard drive.