Unable to Identify Process Causing Server Failure

LinuxRSA · 10-11-2023, 09:01 AM

Hi All

We running various versions of ubuntu and often experience Linux servers freezing & going into a hung state. After the hard reset is completed the server is back online. The issue we having is identifying what processes or applications has utilized high system resources. Our monitoring tool is limited in terms of identifying Process ID's, it however provides the overall CPU MEM Load usage via graphs.

I need to develop a bash script that records Process id's breaching a certain threshold and write the output to /home/monuser/perfstats.txt to identify the PIDS post server crash.

CPU is 95% +
MEM is 95% +
Load is 25 +

The below command is easy and provides the output we need.

Code:

monuser@monitor01:~$ ps -ax -o pid,pcpu,pmem,user --sort -pcpu --no-headers | head
 2033 31.1  3.7 process-name
 4147 21.1  1.0 root
11512 16.0  3.5 process-name
 2481  9.2  0.5 root
 1718  6.7  0.5 root
13285  3.1 29.7 process-name
 1910  0.9  0.2 root
10869  0.8  0.2 root
 3452  0.7  0.1 root
 3185  0.6  0.2 process-name
monuser@monitor01:~$

We know the 2nd Colum is CPU and the 3rd id MEM

The script below is continuous in running and gathering the information throughout the day however we dont require the entries below 95% for CPU and Mem and Load usage.

Code:

#!/bin/bash
# This script monitors CPU Memory and Load. 

while :
do 
  # Check the current PID and usage of CPU and memory and Load.
  cpuandmemUsage=$(ps -ax -o pid,pcpu,pmem,user --sort -pcpu --no-headers | head)
  loadUsage=$(cat /proc/loadavg)

  # Print the usage
  echo "CPU and Mem Usage: $cpuandmemUsage%" >> /home/monuser/perfstats.txt
  echo "SysLoad Usage: $loadUsage MB" >> /home/monuser/perfstats.txt
 
  # Sleep for 1 second
  sleep 1
done

1. Is there a mechanism OR script we can configure to write to a file when that 2 columns value exceeds 95% and activate the script to start recording the data ?

Is this possible.

TIA

syg00 · 10-12-2023, 03:08 AM

Quote:

Originally Posted by LinuxRSA

Is there a mechanism OR script we can configure to write to a file when that 2 columns value exceeds 95% and activate the script to start recording the data ?

Why do you care ?. The disk space used will be trivial.

Bash has arithmetic operators, but they are integer only IIRC. Too hard. Just record the lot and parse it later with something like awk. If you stick the date in the filename it will even create a new file for you each midnight. Any daylight saving issues are your own.

There are other (better) options, but if this is your chosen path, make it as easy as possible.

MadeInGermany · 10-12-2023, 10:04 AM

Code:

while :
do 
  # Check the current PID and usage of CPU and memory and Load.
  cpuandmemUsage=$(ps -ax -o pid,pcpu,pmem,user --sort -pcpu --no-headers | head)
  # Read the top line and split
  read pid pcpu pmem user <<< "$cpuandmemUsage"
  # Get the integer part and compare
  if [ "${pcpu%.*}" -ge 95 ]
  then
    # Print the usage
    read loadUsage < /proc/loadavg
    echo "CPU and Mem Usage: $cpuandmemUsage%"
    echo "SysLoad Usage: $loadUsage MB"
  fi >> /home/monuser/perfstats.txt
 
  # Sleep for 1 second
  sleep 1
done

LinuxRSA · 10-12-2023, 10:39 AM

Quote:

Originally Posted by MadeInGermany

Code:

while :
do 
  # Check the current PID and usage of CPU and memory and Load.
  cpuandmemUsage=$(ps -ax -o pid,pcpu,pmem,user --sort -pcpu --no-headers | head)
  # Read the top line and split
  read pid pcpu pmem user <<< "$cpuandmemUsage"
  # Get the integer part and compare
  if [ "${pcpu%.*}" -ge 95 ]
  then
    # Print the usage
    read loadUsage < /proc/loadavg
    echo "CPU and Mem Usage: $cpuandmemUsage%"
    echo "SysLoad Usage: $loadUsage MB"
  fi >> /home/monuser/perfstats.txt
 
  # Sleep for 1 second
  sleep 1
done

Thanks, will try this.

I have a secondary plan to use my monitoring tool to read the output of a command.

Problem is i cant get to display the output as i need it to display.

This is the output

Quote:

monuser@monitor01:~$ ps -ax -o pid,pcpu,pmem,user --sort -pcpu --no-headers | head
2033 31.1 3.7 process-name
4147 21.1 1.0 root
11512 16.0 3.5 process-name
2481 9.2 0.5 root
1718 6.7 0.5 root
13285 3.1 29.7 process-name
1910 0.9 0.2 root
10869 0.8 0.2 root
3452 0.7 0.1 root
3185 0.6 0.2 process-name
monuser@monitor01:~$

How can i get ps -ax -o pid,pcpu,pmem,user --sort -pcpu --no-headers | head to display only the PID and USER into one column ?

Can we combine the PID and User in the same column?

Something like this

Code:

2033
process-name
4147 
root
11512
process-name

Thanks

syg00 · 10-12-2023, 06:36 PM

I wonder how long it will be before you regret throwing all the historical data away, so you now have no way of doing what-if queries around the time of any event.

uteck · 10-12-2023, 08:30 PM

Have you looked at the sar command? It does a lot of this and more.
https://www.baeldung.com/linux/sar-system-statistics