Hello all,

i've been searching for a few hours/troubleshooting 2 servers that went down today both with kernel panic issues so here are the variables. Both servers are Redhat 5

1) today we were hardening the systems mainly just changing file permissions, or changing ownership of files in specific folders. everything took about 4 hours but once we finished everything was working well. fast forward 2 hours later both displayed full on Kernal panics

2) server room its in is very hot....100F server display does not show any hardware failure...but not convinced. (AC broke)

3) i implemented the same hardening changes on 3 systems with the same image but were VM's.

4) the servers that went down are 2 Physical servers


we really want to get into the kernal logs but can't...booting into single user mode /var is empty. Since its in a secure location i can't just bring in a live CD to do a rescue. Any other ideas? Getting those logs would be my most desired want, or any other ideas to fix


this gave me some ideas what to look for hardware wise http://www.makeuseof.com/tag/dont-pa...kernel-panics/

That probably means you mount /var separately - check /etc/fstab.

What do you mean (exactly) by "single user mode" ?. Does the machine(s) drop into rescue mode on boot, or are you forcing it to boot in single user mode. Be (very) specific.
What distro - again be specific, including release.

Sorry, i'm using Redhat 5, I booted into Single User mode by editing the Kernal file in Grub. init=/bin/bash

so the big question is how do i mount /var in single user mode? i did an initial mount -w -o remount /. can't escalate to root though :/

Why did you do that ?. What is in fstab ?.
Issue a mount command for whatever /var is mounted on. Is it LVM ?. Has a vgchange been issued ?.
We don't have the info you do.

And why did you override init ?. Why do you think the init scripts won't save "yesterdays" logs at boot the same as it always does at boot ?.

the only way i can navigate anything within the system is single user mode. When the system goes through the boot process, everything seems fine and dandy when you get the login on the CLI, after a few seconds the screen goes wack and you see the kernal panic errors with errors such as "cache_fulxarrayx0x74", kme_cace_free, int_check_syscall_exit, drain_arrary, run_workqueue, worker_thread,Child_NP

Is it mount /whatever the path it is in fstab?

how do i get to the init scripts? I'm not fully sure i understand i'm not a redhat admin level by any means and just know how to get around/change settings to make my application work on it

this thread is solved...the issue was Host intrusion Prevention software causing the issue

Care to elaborate on what you did?

sure

step 1: boot into single user mode
setp 2: mounted the directory i needed to get access to /var
step 3: deleted all HIPS related software
step 4: reboot

and everything was working properly

Did you attempt to stop HIPS prior to removing all related software?

Assuming we're talking about McAfee HBSS suit here, CMA is not HIPS. CMA is the service for Policy Auditor. HIPS is a completely separate package and even if you are able to stop HIPS, it will re-start itself if you are in run level 3 or 5. The only sure fire way to stop HIPS is removing the packages associated with it, then rebooting since it has hooks in the kernel.

Moral of the story: get rid of HIPS if you are able to and just use SELinux. Reason being you have more control over the system with SELinux and you can easily troubleshoot any policy violations. If you do continue using HIPS, disable SELinux because if you have both enabled be prepared for an unusable and unstable system.

I wish this was possible but I live in a DoD world.

I understand the struggle, but it is something worth exploring. I don't know what command you're at or what part of the DoD you're under, but it would be a fight worth having to get the policies changed. Most of the paper pushers creating these security policies have no concept of the technical difficulties introduced by HIPS.

no we just wanted to get that shit off, also there was no point of shutting it off if your just going to uninstall, no remenant would remain afterwards