What is an appropriate tool?

At work I have to start monitoring dozens of embedded devices. I need to discern changes in daily configuration backups. The backup files are text files.

Basically I want to create a change log for each device.

I want to automate the monitoring and send differences in daily email reports. I am monitoring only meaningful changes in the backup files. The backup files will be stored on a traditional Linux server.

I am not monitoring changes in a single file but differences between files.

Fundamentally all I need is a diff of changes. I can script that myself using the diff command, but do other tools exist and I am reinventing the wheel?

Other Thoughts

If version or revision control is a candidate, I do not need to track metadata, merges, commit messages, branches, etc.

The backup filenames contain the date and time. Thus each backup file does not overwrite any previous backup.

The date and time also are part of the backup file content. Technically then, each file is different although the configuration might not have otherwise changed. Ideally I do not want to be notified if that is the only difference and do not want anything added to the automated change log.

This thread is related to a previous post, but I thought best to start a new thread rather than continue the older thread.

Thanks again.

(r)syslog
and/or
Monitor System Logs with Logwatch

I probably skipped school on that day. Possibly I am missing the big picture, but I want to monitor differences between two downloaded files, not monitor logs.

diff, yes. And yes, you'll probably need to do some scripting to fine-tune what you want to see.

That's OK, google was in attendance!
How To Compare Two Text Files Using Linux

OK, I'm still not the brightest bulb in the pack, but in my original post I wrote:

I can script that myself using the diff command, but do other tools exist and I am reinventing the wheel?

I thought with that sentence I was declaring I knew how to use the diff command and I could build a script wrapper to diff to compare the files. I am looking for something more broader.

I might have discovered something close: Oxidized, RANCID, rConfig, Unimus.

Looks like Oxidized and RANCID are free/libre. rConfig seems built only for CentOS, and Unimus looks commercial/proprietary. The lead developer with Oxidized is looking for maintainer help, which does not bode well for the future. RANCID requires the expect command, which seems a tad clunky to me (been there done that). Unimus does not seem to support a wide variety of devices yet. Overall, these tools seem somewhat complicated for just downloading and comparing two files.

I found something called cbackup, but after browsing the docs I did not feel warm and fuzzy.

I'll need to dig deeper.

You could use git to track changes, take snapshots of specific configurations, and compare differences or deploy specific configurations.

If you are talking about config files in /etc there is a tool called etckeeper.

https://etckeeper.branchable.com/

this reminds me of a failed (!) undertaking of mine:
http://dt.iki.fi/dotfiles

Why not simplify and just use rsync on /etc/* ???

These are embedded devices. They do not have a traditional FHS tree. Think routers, APs, and the like.

As these are text file dumps, the best I have at the moment is grabbing the most recent two backup files and diff them. Doable but I'm wondering if there are tools that already cater to that kind of monitoring. I found something called graylog, but have not dug deep into that.

Not related but I am using etckeeper on my personal systems.

smallpond suggested a version control system, specifically git, but if you are more familiar with CVS, SVN, or BZR those would work too. Their very reason for existence is to track changes to text files. Configuration files certainly fit that bill at least until some future version of systemd does them in by making them binary. A way to do tracking would be to keep a central repository where they are all stored. The question would be whether to check out / in the files directly to / from the embedded systems or not. I guess that would depend on the complexity of the systems. You could just as well check in and out from a staging machine instead. I've done something similar with CVS back in the day for a server farm of development servers, checking in and out on the servers themselves using CVS, but would probably use git if I were to try that nowadays.

Okay -- version control system (VCS). I am not familiar with implementing at all. ELI5 please.

For example, on our remote logging server I pull the backup file from an embedded device into something I suppose would be called a holding or staging directory? Then perform some kind of commit to the VCS? Then what?

My goal with these backups is not only quick recovery with a rebuild/replacement. I want to monitor changes. First for hackers and second for in-house changes that were not correctly logged.