Hi

I am managing 300+ linux servers and use Ansible/AWX for patching. We are patching all servers monthly.

One division have had one ore two bad experiences with patching everything. Particularly postgres has broken (once or twice) and one time a new kernel broke their software. So they have asked us to apply security patches only.

I have the final say in this and I think it is wiser to patch everything every month and rather fix problems when (if) they happen. We are doing snapshots before patching and removes snapshots 48 hours after patching.

What do you think is best? What are the possible implications of security only patches?

If the security updates are sufficient to keep systems secure I am willing to grant their wish.

Running CentOS and Ubuntu on the servers. We are in the process of migrating to Ubuntu/debian only.

I think the best thing is to only make the changes that keep the systems secure without breaking the software.

Lots of other people agree, and hence there often these things called "security updates" which will keep software secure, but will not change behaviour unless it's absolutely necessary...

I would say: try to apply those (or any) patches on one server and test it. In general you can apply every patch within a release, that should not harm anything (but obviously it is not true, there are cases .....)
From the other hand the installed/needed patches may depend on other software and other requirements. For production servers we always try to avoid any changes, therefore we install only the recommended security patches.

I have a similar issue. 300+ Linux machines. I am on a closed network. I DO NOT apply updates!
Let me explain:
We have a development environment. We develop custom software (and build the hardware to run it). Each software release is assigned a RedHat OS Version. Our software is then developed on this OS. We patch & upgrade on the development system. Until code cut-off. Then the baseline is fixed. We keep this fixed baseline until EOL of the software. All told, Start of Development to End OF Life is about 2 years.

When CVE's come out, most of them do not apply to us -> no action is required -> no patches.
If a CVE appears that _may_ affect us, we look at the risk and the mitigation strategy. Often, no action is required. Sometimes, we can mitigate the risk.
In 20 years, we've actually had to patch ONCE.

You CANNOT just push the updates. If there is a kernel change, and you choose to upgrade to the new kernel, you MUST upgrade the kernel (this may require a re-installation or re-build of some drivers like nVidia). Then REBOOT. Validate everything still works. Then and ONLY THEN can you upgrade all the other packages. Otherwise, you have a descent chance of breaking the machine.

When the new software release comes, a new OS comes with it, which is already patched and hardened.

If at all possible, we DO NOT UPGRADE the computers. We kickstart them. Some of our servers are upgraded. We have a lot of crap on these machines that doesn't need to be there anymore, but no one knows for sure if any of the dependencies are still in use. So we leave them until the hardware is EOL'd. *YUCK*

When we do push updates, patches, new software (in our development environment), we do it piece meal. A few workstations. Then a bit more workstations. Then a few servers -- not all the same type of servers.
Ex: We won't do all of our webserver. Just one or two. On the same day, we might do one /home server, 2 database servers, etc.
This way, if something creates a SNAFU, it is a minimal impact. We can work it out before continuing.
With out 300+ hosts, we might make changes every 2 weeks. It could take 3 months to push our updates to all of the machines.

Having a cron that updates all of the packages once a week will cause headaches and frustrations and unnecessary down time.

Do it smartly. Only do what you need, when you need it.
If you are on a closed network, you most likely don't need it all.

/rant