Skip to content

Commit

Permalink
fix: Prevent Gunicorn child workers hangup
Browse files Browse the repository at this point in the history
Problem:

- Prerequisite: long running reqeust is going on.
- `bench restart` is requested.
- supervisord sends sigterm to gunicorn master process.
- gunicorn passes it to all child table and waits for graceful shutdown (30 seconds)
- supervisord has no chill, and sends sigkill to master process in 10 seconds
- gunicorn master proesss is dead, so now child workers will keep running until they complete request which can take a really long time.
- This entire time the sites will be down.

Fix:
- Explicitly encode default graceful_timeout in config - 30 seconds.
- Make supervisor wait 10 more seconds for gunicorn to do its thing, then only send sigkill.
  • Loading branch information
ankush committed Nov 2, 2023
1 parent 53a8fed commit 36f2194
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion bench/config/templates/supervisor.conf
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,15 @@
; priority=1 --> Lower priorities indicate programs that start first and shut down last
; killasgroup=true --> send kill signal to child processes too

; graceful timeout should always be lower than stopwaitsecs to avoid orphan gunicorn workers.
[program:{{ bench_name }}-frappe-web]
command={{ bench_dir }}/env/bin/gunicorn -b 127.0.0.1:{{ webserver_port }} -w {{ gunicorn_workers }} --max-requests {{ gunicorn_max_requests }} --max-requests-jitter {{ gunicorn_max_requests_jitter }} -t {{ http_timeout }} frappe.app:application --preload
command={{ bench_dir }}/env/bin/gunicorn -b 127.0.0.1:{{ webserver_port }} -w {{ gunicorn_workers }} --max-requests {{ gunicorn_max_requests }} --max-requests-jitter {{ gunicorn_max_requests_jitter }} -t {{ http_timeout }} --graceful-timeout 30 frappe.app:application --preload
priority=4
autostart=true
autorestart=true
stdout_logfile={{ bench_dir }}/logs/web.log
stderr_logfile={{ bench_dir }}/logs/web.error.log
stopwaitsecs=40
user={{ user }}
directory={{ sites_dir }}

Expand Down

0 comments on commit 36f2194

Please sign in to comment.