How to verify if the node reboot was triggered by software watchdog?

msutic · 07-12-2022, 09:17 AM

Hello,

I have created software watchdog using command:

Code:

$ sudo modprobe softdog soft_margin=60

And in OS logs I have seen message:

Code:

[    3.757002] softdog: Software Watchdog Timer: 0.08 initialized. soft_noboot=0 soft_margin=60 sec soft_panic=0 (nowayout=0)

When I trigger node reboot using command below, node reboots, but there is no information in the log that softdog triggered reboot:

Code:

echo a | sudo tee /dev/watchdog

If software watchdog is created using soft_noboot=1 option, just to verify softdog without actual reboot, then information is logged:

Code:

softdog: Triggered - Reboot ignored

Based on the softdog implementation we should have log:
https://github.com/spacex/kernel-cen...hdog/softdog.c

Code:

static void watchdog_fire(unsigned long data)
	{
		if (soft_noboot)
			pr_crit("Triggered - Reboot ignored\n");
		else if (soft_panic) {
			pr_crit("Initiating panic\n");
			panic("Software Watchdog Timer expired");
		} else {
			pr_crit("Initiating system reboot\n");
			emergency_restart();
			pr_crit("Reboot didn't ?????\n");
		}
	}

OS: CentOS Linux release 7.9.2009 (Core)
Linux test1 3.10.0-1160.62.1.el7.x86_64 #1 SMP Tue Apr 5 16:57:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

It looks like node was rebooted before log info was written to disk.
Can I somehow verify that node was rebooted by software watchdog?

Thank you

smallpond · 07-20-2022, 08:02 PM

also check what is reported by the "last" command for reboot reason.

msutic · 07-21-2022, 01:18 AM

Quote:

Originally Posted by smallpond

also check what is reported by the "last" command for reboot reason.

With "last" command I can find when reboot happened and was it graceful, but cannot find the reason.

I have inspected audit logs and configured persistent systemd journal with journalctl but still no information that watchdog/softdog triggered reboot.

pan64 · 07-21-2022, 02:29 AM

At first I would try to check if the triggered reboot will create any log event anywhere. And you will know if that last reboot was really triggered by it or something else happened.

msutic · 07-22-2022, 01:50 AM

Quote:

Originally Posted by pan64

At first I would try to check if the triggered reboot will create any log event anywhere. And you will know if that last reboot was really triggered by it or something else happened.

When I trigger reboot manually, intentionally writing something to "/dev/watchdog" device I see message "watchdog: watchdog0: watchdog did not stop!".
But, when watchdog triggers restart when node is overloaded or when used with Patroni cluster (timer not updated) then there is no any info.

This looks as a bug to me. If "Initiating system reboot" is supposed to be logged before "emergency_restart()" then I would expect this info - at least when node is idle.
Otherwise what is the purpose of this log message?!