[SOLVED] Raspbian (Stretch): systemd not starting nginx

Today was supposed to be "Get Certbot Up And Running Day" but it's turned into yet another systemd headache. [sigh]

The topic is a good summary: When I boot an RPi on our network, the nginx service does not start. All of the service files being used are unchanged (by me) from the default Stretch installation---except for any subsequent updates.

I've done:

# systemctl enable nginx.service

and "chkconfig" shows that it's "on" but, following a reboot, I see no nginx processes. If I check the service's status, I get:

# systemctl status nginx.service
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sat 2020-07-11 12:22:52 CDT; 7min ago
     Docs: man:nginx(8)
  Process: 472 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)

Jul 11 12:22:51 vger2 systemd[1]: Starting A high performance web server and a reverse proxy server...
Jul 11 12:22:52 vger2 nginx[472]: nginx: [emerg] host not found in upstream "testsite.mydomain.net:80" in /etc/nginx/sites-enabled/testmydomainnet.conf:5
Jul 11 12:22:52 vger2 nginx[472]: nginx: configuration file /etc/nginx/nginx.conf test failed
Jul 11 12:22:52 vger2 systemd[1]: nginx.service: Control process exited, code=exited status=1
Jul 11 12:22:52 vger2 systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Jul 11 12:22:52 vger2 systemd[1]: nginx.service: Unit entered failed state.
Jul 11 12:22:52 vger2 systemd[1]: nginx.service: Failed with result 'exit-code'.

I can successfully start the process by hand using:

# /etc/init.d/nginx start

or

# systemctl start nginx.service

Either way I start it, I can verify that the service is up and running:

# systemctl status nginx.service
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2020-07-11 12:31:07 CDT; 26s ago
     Docs: man:nginx(8)
  Process: 842 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
  Process: 839 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
 Main PID: 843 (nginx)
    Tasks: 5 (limit: 4915)
   CGroup: /system.slice/nginx.service
           ├─843 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
           ├─844 nginx: worker process
           ├─845 nginx: worker process
           ├─846 nginx: worker process
           └─847 nginx: worker process

Jul 11 12:31:06 vger2 systemd[1]: Starting A high performance web server and a reverse proxy server...
Jul 11 12:31:07 vger2 systemd[1]: Started A high performance web server and a reverse proxy server.

Thinking that it was complaining that the test server was not running apache at the time, from the RPi, I verified that it was:

# nmap testsite.mydomain.net
Starting Nmap 7.40 ( https://nmap.org ) at 2020-07-11 12:32 CDT
Nmap scan report for testsite.mydomain.net (192.168.13.7)
Host is up (0.00013s latency).
Not shown: 996 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
80/tcp   open  http
111/tcp  open  rpcbind
2049/tcp open  nfs
MAC Address: D4:3D:7E:DF:2D:45 (Micro-Star Int'l)

Nmap done: 1 IP address (1 host up) scanned in 2.31 seconds

That server is pingable by name so it does not appear to be a DNS problem. I also can reach the test web server -- once nginx has been started by hand -- using any browser I point at it.

The nginx configuration seems healthy:

# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# /etc/init.d/nginx testconfig
[ ok ] Testing nginx configuration:.

If I take a look at the systemd logging ("journalctl -xe"), I can see where the first attempt, at boot, failed and where the manual startup succeeded.

Though I'm not sure why the nginx startup is behaving differently at boot than it is when I'm at a command prompt, I though debugging nginx might provide some insight. The Raspbian nginx was compiled with debug support so I added a little config file in /etc/nginx/conf.d containing:

error_log /var/log/nginx/debugging.log debug;

and reloaded the nginx config. I have a debugging.log file but restarting the nginx service or rebooting never places anything in it: still sitting there with 0 bytes.


Anything else I should be checking to see why the systemd is having trouble starting this service?

TIA...

Originally Posted by rnturn Today was supposed to be "Get Certbot Up And Running Day" but it's turned into yet another systemd headache. [sigh] . . . Though I'm not sure why the nginx startup is behaving differently at boot than it is when I'm at a command prompt, I though debugging nginx might provide some insight. The Raspbian nginx was compiled with debug support so I added a little config file in /etc/nginx/conf.d containing: error_log /var/log/nginx/debugging.log debug; and reloaded the nginx config. I have a debugging.log file but restarting the nginx service or rebooting never places anything in it: still sitting there with 0 bytes.

Debugging was a bust. I've made the debugging config file hidden (for later use).

It was suggested (elsewhere) that nginx might be starting up too early and that, I'm assuming since the nginx config files were defining systems using FQDNs instead of IP addresses, that the service file needed to wait for some other service (probably bind9) to finish startup. I took the approach that nginx should just wait for everything needed for the multi-user target and modified nginx's service contain:

[Unit]
#After=network.target
After=multi-user.target

[Install]
#Wants=multi-user.target

and it now starts without problems at boot. Until, that is, some future systemd change comes along to make things, um, interesting again.

Later...

Kudos. Please mark this thread as [solved], since you solved the problem .