2 PC Linux server, hangs with kernel panic.

Stilltray · 11-24-2007, 06:53 AM

I have a problem with 2 RedHat installed Linux PC's.
Oracle cluster software is also installed on both machines. They both work on (the same) SAN.

Problem is that weekly, at exactly the same time the server, so both PCs, hangs.
The only thing I see is kernel panic on both PC's (--> flashing lights on the keyboards).

The only thing I can do is manual rebooting the machines...

After reading several articles on this forum I read 2 things:

1:
dmesg > /tmp/kernel.panic.txt

should give me info on the reason what the kernel panic causes? Can I use this after manual rebooting?

And

2:
To see what is happening at the very last moment before the server hangs use the top command.
How can I log ALL info from top in a file, so I maybe can see what's happening just before the server freezes?

Anybody any experience with these possibilities

Could the hangcheck-timer be the reason for freezing the server?

Thanks for helping. I've no ideas anymore...

harry edwards · 11-24-2007, 03:37 PM

It would be good if you could include the kernel panic information. A few things to check:

a) Are you on the latest kernel for your distro - you may have hit a kernel bug.

b) Are your Oracle databases patched to the latest patch-set as recommend by Oracle - Oracle may have hit a bug.

c) Does the Oracle alert log show anything of use?

If you are stuck after doing the above you may need to log this as bug within your Distro bug tracking system.

Stilltray · 11-24-2007, 04:10 PM

It's by far not the latest kernel. But this kernel is used for more than one year now.
All that time it has also the same Oracle version.

The alertlogfiles etc don't show anything what could be a problem.

Because both PC's hang, there's no logging what's going on. Not from Oracle, neither from Linux.

Is there anything logged before kernel panic?
Can I see it after rebooting in a file or watsoever?
(I know there's a crash utility for crash dumps, but I think I've not enough experience to use it at this moment)

harry edwards · 11-24-2007, 04:30 PM

I know from experience that a bug can suddenly arise even after running Oracle for a year. It maybe worth patching at some point if you cannot fix the problem.

I'm have some experience tracing kernel panics but not a lot. Here's a few links to try:

Kernel.org Bug Tracking - Have you searched here?

http://bugzilla.kernel.org/

Kernel Panic Troubleshooting

http://rhcelinuxguide.wordpress.com/.../linux-kernel/