systemd and Linux Services Explained: Managing Background Processes

Nearly every modern Linux distribution uses systemd to manage background services — understanding it directly replaces a lot of "restart it and hope" troubleshooting with actually reading what went wrong.

The core commands

systemctl start nginx        # start now (this session only)
systemctl stop nginx          # stop now
systemctl restart nginx       # stop then start
systemctl status nginx        # current state + recent log lines
systemctl enable nginx        # start automatically on future boots
systemctl disable nginx       # don't start automatically anymore
systemctl enable --now nginx  # enable AND start in one command

The enable vs. start distinction is the single most common point of confusion — start alone doesn't survive a reboot, and enable alone doesn't affect the current session.

A minimal unit file

A unit file defines a service — what command to run, when to start it, and how to restart it on failure:

# /etc/systemd/system/my-app.service
[Unit]
Description=My background application
After=network.target
 
[Service]
ExecStart=/usr/bin/node /opt/my-app/server.js
Restart=on-failure
RestartSec=5
User=myappuser
Environment=NODE_ENV=production
 
[Install]
WantedBy=multi-user.target

After creating or editing a unit file, systemd needs to be told to re-read it:

systemctl daemon-reload
systemctl enable --now my-app

The three sections, briefly

• [Unit] — metadata and ordering: a description, and After=/Requires= to control what this service should start after (or depend on).
• [Service] — the actual behavior: what command runs (ExecStart), what to do on failure (Restart=on-failure), which user runs it, and environment variables.
• [Install] — how this unit integrates with enable — WantedBy=multi-user.target is the standard choice for a normal background service that should start at boot.

Reading logs with journalctl

systemd captures a service's stdout/stderr automatically — journalctl is how you read it, and it's almost always the actual answer to "why did my service fail":

journalctl -u my-app                # all logs for this unit
journalctl -u my-app -f              # follow in real time, like tail -f
journalctl -u my-app --since "1 hour ago"
journalctl -u my-app -n 50           # last 50 lines

systemctl status shows a truncated preview of recent log lines — when that's not enough context, journalctl -u <service> shows the complete picture.

Automatic restart on failure

Restart=on-failure (in the unit file above) is what keeps a crashed service running without manual intervention — but it's worth pairing with limits, since an unbounded restart loop on a genuinely broken service just burns CPU repeatedly failing:

[Service]
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=60
StartLimitBurst=3

StartLimitBurst=3 within StartLimitIntervalSec=60 means systemd gives up (rather than looping forever) after 3 failed restarts within 60 seconds — a genuinely broken service fails loudly and stays stopped, rather than consuming resources in an infinite crash loop.

Timers: systemd's alternative to cron

systemd can also schedule recurring jobs, as a modern alternative to a crontab entry — pairing a .service unit (what to run) with a .timer unit (when to run it):

# /etc/systemd/system/backup.timer
[Timer]
OnCalendar=daily
Persistent=true
 
[Install]
WantedBy=timers.target

systemctl enable --now backup.timer
systemctl list-timers  # see all scheduled timers and their next run

Persistent=true means a missed run (the machine was off at the scheduled time) executes as soon as the system is back up — a real advantage over plain cron, which simply skips a run the machine wasn't on for.

Checking why a service failed to start

systemctl status reports the immediate failure reason, but for a genuinely unclear failure, checking the exit code and the full log together usually explains it:

systemctl status my-app
# Look for: "Active: failed (Result: exit-code)" and the "Main PID" exit status
 
journalctl -u my-app -n 100 --no-pager

A service failing immediately on every start attempt (rather than running for a while first) is very often a configuration or permissions issue — a wrong path in ExecStart, a User= that doesn't have permission to bind the configured port, or a missing environment variable the application expects.

Reloading config without a full restart

Some services support reloading their configuration without dropping active connections — systemctl reload sends the service a signal (typically SIGHUP) it can handle to re-read config, rather than the harder stop-then-start of a full restart:

systemctl reload nginx        # nginx re-reads config, no dropped connections
systemctl reload-or-restart my-app  # reload if supported, else restart

Not every service supports this — it depends on the application itself handling the reload signal — but for ones that do (nginx being the canonical example), it's meaningfully less disruptive than a full restart for a routine config change.

Sockets: starting a service on demand

systemd can also start a service lazily, the first time something connects to its configured socket, rather than always running in the background:

# my-app.socket
[Socket]
ListenStream=8080
 
[Install]
WantedBy=sockets.target

systemctl enable --now my-app.socket

The service itself only starts when the first connection actually arrives on port 8080 — useful for infrequently-used services where running constantly in the background would be wasted resources, at the cost of a small startup delay on that first connection.

Common mistakes

Related reading

• Essential Linux Commands Every Developer Should Know — shares tags: linux, devops (same category).
• Understanding File Permissions in Linux (chmod, chown Explained) — shares tags: linux.
• Docker Compose for Local Development Environments — shares tags: devops.
• CI/CD Pipelines Explained: From Commit to Production — shares tags: devops.
• Infrastructure as Code: Why Terraform Won — shares tags: devops.