NetworkManager + firewalld + /etc/sysconfig/network-scripts = not working together
 

I've made three CentOS 7 installation attempts to configure a simple firewall/router box with 2 nics.
I got myself into a seemingly circular scenario where NetworkManager and firewalld and /etc/sysconfig/network-scrpts/ifcfg-***** were interfering or overwriting each other.

I would need to ifdown enp3s7 the internal LAN nic in order to make the external internet enp2s0 reach websites and ping nameservers.
Or after completing firewall-cmd --complete-reload the internal LAN nic would still provide private ip addresses via dhcpd server but LAN clients could not access the internet.

After searching, I found some others experiencing similar adventures: https://github.com/t-woerner/firewalld/issues/195 ((this person's ability to analyze multiple variables and how they're working together is far superior to mine))

I also found more than a few sysadmin explanations that relied on disabling NetworkManager altogether to make network-scripts/ifcfg-***** coordinate with firewalld services.

I'm hoping the following info is helpful to others struggling with the same problem:

1. sysctemctl stop NetworkManager

2. systemctl disable NetworkManager

3. Create dhcp ifcfg-***** for external interface. It must include a “ZONE=external” statement even though firewalld service will overwrite and erase it like this “ZONE=”
Example (external/internet nic):
4. Create static ip address ifcfg-enp3s7 for internal interface.
Example (internal/LAN nic):
5. As said in #3, firewalld will erase the ZONE setting on the external nic configured for dhcp.
The only way I've found to deal with this overwriting is to make the intended external ethernet device associated with the default zone in firewalld. When firewalld reads the empty zone reference "ZONE=____" it will revert and assign the default zone I set like this ---
6. The external ethernet device won’t work (cannot ping any internet host) until you manually Deactivate it and then Reactivate it.
~# ifdown enp2s0
~# ifup enp2s0

I didn't include my dhcpd server settings or firewalld settings for brevity.
Please let me know if those would be helpful.
I'm still trying to learn all the nuances of making systemd, selinux, firewalld, iptables, and NetworkManager work together, so if you can lend any guidance to improve my current configuration I'd be very thankful.
For now, NetworkManager will remain disabled on my routerbox.

Forgot to mention, after reboot, you'll need to manually re-set the external internet nic as shown above.
Definitely not ideal for anyone relying on remote ssh access.
Local access to terminal required to re-set the external internet device.

Made another try.
I'll need to do some remote management so it's not acceptable to potentially get locked out upon reboot.

On latest attempt, 1. start NetworkManager and enable NetworkManager

The statement above is wrong. ZONE= statement is not necessary to make ethernet device settings work with firewalld.
Also wrong, firewalld is not overwriting anything in /etc/sysconfig/network-scripts/ifcfg-*****.
I removed all previous ifcfg-***** scripts and created new ones using a combination of the nmcli and nmtui commandline tools for controlling NetworkManager devices.

External ethernet device:
Internal (LAN) ethernet device:
I believe the cause of all the previous problems with needing to deactivate the internal ethernet device in order to reach internet hosts and ping dns on the external ethernet nic was due to setting DEFROUTE=yes on the internal (LAN) device. Upon changing to DEFROUTE=no on the internal nic, all routing across from LAN to Internet and Internet to LAN works upon reboot.

I tried to provide a GATEWAY setting on the internal ethernet device using nmtui but it doesn't write to ifcfg-internal.
I manually inserted GATEWAY=10.10.1.1 into ifcfg-internal but it gets removed by NetworkManager upon reboot.

systemctl status NetworkManager also reports --- "GATEWAY will be ignored when DEFROUTE is disabled." Okay. NetworkManager doesn't need it to forward packets from LAN clients.

Again, I misunderstood. It's not necessary to insert ZONE statements into the ethernet device configuration.
Assign ethernet device zones using firewall-cmd.
Configure zone parameters also using firewall-cmd.

selinux is enabled and enforcing.
firewalld is enabled and directing iptables filter and NAT.
NetworkManager is controlling ethernet devices configuration and active connections.
dhcpd is leasing ip addresses to LAN clients.

Everything works upon update and reboot.
SOLVED