[SOLVED] Trying to filter lines in a log from last hour

skagnola · 04-01-2022, 03:06 PM

Hello!

Looking for some help trying to work with a log file in a CentOS 7.9 machine. I would like to find entries in the log within the past hour. Then I will grep the lines for specific dialog.

The log timestamps are like below

Code:

[2022/04/01 14:27:46.0277603]

I am running into an issue, where trying the below grep for the date range is coming back with nothing. This is the part that needs the halp! :-)

Code:

grep "^(date -d -1hour +'%Y/%m/%d')" /var/log/this.log

If the date grep works, then I will pass another grep

Code:

| grep -i blabeedeebla

Ultimately I will put this in a bash script that will email out a notice if it finds the 'blabeedeebla'.

dugan · 04-01-2022, 04:24 PM

Code:

journalctl --since "1 hour ago"

skagnola · 04-01-2022, 04:40 PM

Quote:

Originally Posted by dugan

Code:

journalctl --since "1 hour ago"

Hey, dugan! Thanks for the quick response. Unfortunately, this service isn't sending messages to the journal. :-(

Running that will definitely spit stuff out, but nothing from the service that writes to this particular log file.

boughtonp · 04-01-2022, 04:50 PM

Quote:

Originally Posted by skagnola

The log timestamps are like below

Code:

[2022/04/01 14:27:46.0277603]

I am running into an issue, where trying the below grep for the date range is coming back with nothing. This is the part that needs the halp! :-)

Code:

grep "^(date -d -1hour +'%Y/%m/%d')" /var/log/this.log

There are two issues with this.

Firstly, you're assuming the date starts at the start of the string, but there's an opening bracket there - to fix that, you can either remove the "^" or start the expression with "^\["

The second issue is that you're missing the $ that will cause your date command to be evaluated, i.e: "$(date -d -1hour +'%Y/%m/%d')"

boughtonp · 04-01-2022, 05:08 PM

I thought there might also be a way to do this with Awk, but its date/time functionality is crude and makes it a bit messy...

Code:

$ DummyLogs='[2022/04/01 14:27:46.0277603] line 1
[2022/03/31 14:27:46.0277603] line 2'

$ awk -vCutOff="$(date -d -10hour +'%s')" '{ d=$1" "$2 ; gsub(/[^0-9]/," ",d) ; if (mktime(d)>CutOff){print}}' <<< "$DummyLogs"
[2022/04/01 14:27:46.0277603] line 1

syg00 · 04-01-2022, 08:18 PM

Simple enough to add the logfile to the journal. If it ain't your code, simply use systemd-cat - a reasonable write-up is here

skagnola · 04-01-2022, 10:33 PM

Thanks, Peter! That got me further along. It is now showing the current day of log entries from midnight on, however. Not sure why it is not picking up just the last hour arg. This is what I am using now

Code:

grep "$(date -d -1hour "+%Y/%m/%d") /var/log/file.log

Tried a few variations on the -d => "1 hour ago", -1hour, -60min, ... not sure if it is a version of 'date' going on?

I have not tried the awk method yet.

syg00 - I will definitely give that a shot as well, feeding the log to the journal!

boughtonp · 04-02-2022, 07:46 AM

Quote:

Originally Posted by skagnola

It is now showing the current day of log entries from midnight on, however. Not sure why it is not picking up just the last hour arg.

Run your date command on its own and the first past of the reason would hopefully be obvious: you've specified an output pattern that only consists of year/month/day.

However, you need to note that even if you add time/hours, regex/grep doesn't have a concept of "greater than". You need something like Awk/Perl/etc to achieve that logic (or a specialized tool).

Quote:

syg00 - I will definitely give that a shot as well, feeding the log to the journal!

Again, if that's an option you should do it, then you get a whole bunch of useful and tested functionality you can use.

But, I wouldn't automatically do this for all log files - web server request logs are an example of something I wouldn't want in there.

Still, depending on what task you're trying to achieve, there are other log parsing/management tools that may help you do it easier.

skagnola · 04-04-2022, 08:46 AM

Appreciate all the help with this!

I will try to see how things go with feeding just this particular set of logs to the journal. I figured there were reasons why the app dev didn't want these sent to the journal, but it may worth trying it out. Sure beats crafting the customized regex etc. to get what you need out from the raw log.

Thanks again, guys!