fix: Prevent Gunicorn child workers hangup #1493

ankush · 2023-11-02T06:16:01Z

Problem:

Prerequisite: long running reqeust is going on.
bench restart is requested.
supervisord sends sigterm to gunicorn master process.
gunicorn passes it to all child workers and waits for graceful shutdown (30 seconds)
supervisord has no chill, and sends sigkill to master process in 10 seconds
gunicorn master proesss is dead, so now child workers will keep running until they complete request which can take a really long time.
This entire time the sites will be down.

Fix:

Explicitly encode default graceful_timeout in config - 30 seconds.
Make supervisor wait 10 more seconds for gunicorn to do its thing, then only send sigkill.

Detailed analysis: https://gameplan.frappe.cloud/g/engineering/projects/152/discussion/2523

Problem: - Prerequisite: long running reqeust is going on. - `bench restart` is requested. - supervisord sends sigterm to gunicorn master process. - gunicorn passes it to all child table and waits for graceful shutdown (30 seconds) - supervisord has no chill, and sends sigkill to master process in 10 seconds - gunicorn master proesss is dead, so now child workers will keep running until they complete request which can take a really long time. - This entire time the sites will be down. Fix: - Explicitly encode default graceful_timeout in config - 30 seconds. - Make supervisor wait 10 more seconds for gunicorn to do its thing, then only send sigkill.