The aim

I want to have a Linux service that is:

  • automatically started on boot,
  • if the script start is unsuccessful, it will retry infinitely,
  • whenever the service file is changed, the service should be restarted.

High Level solution

  1. We will use systemd service configured to restart upon failure,
  2. We will use another systemd service (a “re-starter” service) that will be triggered whenever file changes.

Detailed steps

Create a service

For this example we’ll use a Python script that starts a simple web server. I am using Flask but the actual content of the file is not relevant here.
The script is saved in /opt/my-service/server.py.

Now we need to create a systemd service file. The file should be saved in /etc/systemd/system/my-service.service with the following content:

[Unit]
Description=My Service
After=network.target

[Service]
WorkingDirectory=/opt/my-service/
ExecStart=/opt/my-service/venv/bin/python /opt/my-service/server.py
Restart=always
RestartSec=5
Environment=FLASK_ENV=production

[Install]
WantedBy=multi-user.target

I am using the Python virtual environment to run the service. The Restart=always and RestartSec=5 will make sure that the service is restarted every 5 seconds, infinitely.

Now we need to reload systemd daemon, start and enable the service:

sudo systemctl daemon-reload
sudo systemctl start my-service.service
sudo systemctl enable my-service.service

start - means it starts the service immediately,
enable - makes sure it starts on boot,

We can check the status of the service with:

systemctl status my-service.service

and verify that it is running:

● my-service.service - My Service
  Loaded: loaded (/etc/systemd/system/my-service.service; enabled; preset: enabled)
  Active: active (running) since Thu 2024-07-25 00:53:35 CEST; 11min ago
Main PID: 21345 (python)
   Tasks: 4 (limit: 76784)
  Memory: 27.5M (peak: 28.2M)
     CPU: 185ms
CGroup: /system.slice/my-service.service
          ├─21345 /opt/my-service/my-service/venv/bin/python /opt/my-service/my-service/server.py
          ├─21347 /opt/my-service/my-service/venv/bin/python /opt/my-service/my-service/server.py
          ├─21348 /opt/my-service/my-service/venv/bin/python /opt/my-service/my-service/server.py
          └─21349 /opt/my-service/my-service/venv/bin/python /opt/my-service/my-service/server.py

lip 25 00:53:35 desktop systemd[1]: Started my-service.service - My Service
lip 25 00:53:35 desktop python[21345]: [2024-07-25 00:53:35 +0200] [21345] [INFO] Starting gunicorn 22.0.0
lip 25 00:53:35 desktop python[21345]: [2024-07-25 00:53:35 +0200] [21345] [INFO] Listening at: http://127.0.0.1:7777 (21345)
lip 25 00:53:35 desktop python[21345]: [2024-07-25 00:53:35 +0200] [21345] [INFO] Using worker: sync
lip 25 00:53:35 desktop python[21347]: [2024-07-25 00:53:35 +0200] [21347] [INFO] Booting worker with pid: 21347
lip 25 00:53:35 desktop python[21348]: [2024-07-25 00:53:35 +0200] [21348] [INFO] Booting worker with pid: 21348
lip 25 00:53:35 desktop python[21349]: [2024-07-25 00:53:35 +0200] [21349] [INFO] Booting worker with pid: 21349

Let’s also show the logs of the service and keep it somewhere nearby:

journalctl -u my-service.service -f

So now we have a service that is started on boot and will be restarted every 5 seconds if it fails.

Create a re-starter

Systemd has a pretty neat feature called Path unit configuration. It allows to monitor a file (or directory) and trigger a service whenever the file changes.

There is one important caveat - whenever the file changes, the service will be started. Not restarted.

Hence, we cannot just simply add a Path unit configuration (my-service.path file) to monitor our service.py and think it will restart the service.

We need to do a bit of trickery here; the idea is to:

  • create a re-starter service (my-service-restarted.service) that, when executed, will simply restart the my-service.service
  • make our re-starter service be triggered (i.e. started) whenever the server.py file changes.

Create a re-starter service

Let’s start with the re-starter service. The file should be saved in /etc/systemd/system/my-service-restarter.service and is quite simple:

[Unit]
Description=My Service Restarter
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/bin/systemctl restart my-service.service

[Install]
WantedBy=multi-user.target

As you can see - it has ExecStart that… restarts the my-service.service. So despite how weird it sounds, it’s like:

If this my-service-restarter starts, it will restart my-service.

Create a re-starter path unit configuration

Now we need to create a Path unit configuration that will monitor the server.py file and trigger the my-service-restarter.service whenever the file changes.

The file should be saved in /etc/systemd/system/my-service-restarter.path and should have the following content:

[Unit]
Description=My Service Restarter Watcher

[Path]
PathChanged=/opt/my-service/server.py

[Install]
WantedBy=multi-user.target

PathChanged means that the systemd (using inotify under the hood) will trigger my-service-restarter.service whenever the file server.py has been changed and the file has been closed.

If you’re interested in every single change triggering the service, you should use PathModified instead.

Remember about reloading and enabling of the services:

sudo systemctl daemon-reload
sudo systemctl enable my-service-restarter.path

Test

Now the test is quite simple:

  • Keep the journalctl -u my-service.service -f running and,
  • change the server.py file.

You should see the service being restarted whenever you make some changes and close the file.

Useful resources