Question

systemd program falls asleep and only wakes up again when I connect to the droplet via SSH

I have a simple Symfony application hosted on a DigitalOcean Droplet. It is a private application with a password-protected dashboard (apache) and there are a few background processes. That’s why I chose the following droplet for now:

  • Premium Intel
  • 1 vCPU
  • 1 GB Memory
  • 35 GB SSD
  • 1 TB Transfer

The background processes run via the Symfony Messenger. I have introduced the following systemd program for this:

[Unit]
Description=Symfony messenger-consume %i (async)

[Service]
ExecStart=php /var/www/html/current/bin/console messenger:consume async -vv --time-limit=3600 --memory-limit=372M
Restart=always
RestartSec=30
WorkingDirectory=/var/www/html/current
StandardOutput=append:/var/www/html/shared/var/log/systemd/messenger-async.log
StandardError=append:/var/www/html/shared/var/log/systemd/messenger-async.log

[Install]
WantedBy=default.target

The –time-limit=3600 option terminates the program after one hour and then restarts it automatically. With the option –memory-limit=372M this also happens when the memory has been reached.

The configured log file shows that this works:


 [OK] Consuming messages from transport "async".                                

 // The worker will automatically exit once it has exceeded 372M of memory, been
 // running for 3600s or received a stop signal via the messenger:stop-workers  
 // command.                                                                    

 // Quit the worker with CONTROL-C.                                             

19:42:25 INFO      [messenger] Stopping worker. ["transport_names" => ["async"]]
19:42:25 INFO      [messenger] Worker stopped due to time limit of 3600s exceeded ["timeLimit" => 3600]

 [OK] Consuming messages from transport "async".                                

 // The worker will automatically exit once it has exceeded 372M of memory, been
 // running for 3600s or received a stop signal via the messenger:stop-workers  
 // command.                                                                    

 // Quit the worker with CONTROL-C.                                             

20:42:57 INFO      [messenger] Stopping worker. ["transport_names" => ["async"]]
20:42:57 INFO      [messenger] Worker stopped due to time limit of 3600s exceeded ["timeLimit" => 3600]

...

After some time, the program is terminated because it seems to receive a signal 15 from somewhere:

 [OK] Consuming messages from transport "async".                                

 // The worker will automatically exit once it has exceeded 372M of memory, been
 // running for 3600s or received a stop signal via the messenger:stop-workers  
 // command.                                                                    

 // Quit the worker with CONTROL-C.                                             

22:18:27 INFO      [messenger] Received signal 15. ["signal" => 15,"transport_names" => ["async"]]
22:18:27 INFO      [messenger] Stopping worker. ["transport_names" => ["async"]]

However, the program will no longer restart automatically after this. Only when I reconnect to the server via SSH does the program start again automatically. Does anyone have any idea what could be causing my program to terminate and why it only restarts when I connect to the server? I can’t see anything in the graphs of the droplet, for example there is no utilization above 75%.


Submit an answer


This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

KFSys
Site Moderator
Site Moderator badge
July 17, 2024

Heya,

I know you’ve figured it out but here are some other ideas I think might help. Firstly, try Service Watchdog.

Enable WatchdogSec in the systemd service file to ensure the service is monitored and restarted if it becomes unresponsive.

[Service] 
WatchdogSec=3600

Add Restart on-failure Policy: Ensure that the systemd service has a proper restart policy to handle unexpected terminations.

[Service] 
Restart=on-failure

Here’s an updated version of your systemd service file with some additional configurations:

[Unit]
Description=Symfony messenger-consume %i (async)
After=network.target

[Service]
User=www-data
Group=www-data
ExecStart=php /var/www/html/current/bin/console messenger:consume async -vv --time-limit=3600 --memory-limit=372M
Restart=on-failure
RestartSec=30
WorkingDirectory=/var/www/html/current
StandardOutput=append:/var/www/html/shared/var/log/systemd/messenger-async.log
StandardError=append:/var/www/html/shared/var/log/systemd/messenger-async.log
WatchdogSec=3600

[Install]
WantedBy=multi-user.target

Problem found. I just noticed that the program doesn’t just exit after a certain time, but directly after logout. And that’s how I came up with this solution: https://unix.stackexchange.com/questions/521538/system-service-running-as-user-is-terminated-on-logout

Try DigitalOcean for free

Click below to sign up and get $200 of credit to try our products over 60 days!

Sign up

Become a contributor for community

Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

DigitalOcean Documentation

Full documentation for every DigitalOcean product.

Resources for startups and SMBs

The Wave has everything you need to know about building a business, from raising funding to marketing your product.

Get our newsletter

Stay up to date by signing up for DigitalOcean’s Infrastructure as a Newsletter.

New accounts only. By submitting your email you agree to our Privacy Policy

The developer cloud

Scale up as you grow — whether you're running one virtual machine or ten thousand.

Get started for free

Sign up and get $200 in credit for your first 60 days with DigitalOcean.*

*This promotional offer applies to new accounts only.