@Marvin How to activate the function

handle_sigterm

to gracefully exit a worker?

To activate the

handle_sigterm

function for gracefully shutting down a worker in Prefect, you need to register the signal handler for SIGTERM within your task worker. Here's a brief guide on how it works: 1. Definition: The

handle_sigterm

function is designed to handle the SIGTERM signal, initiating a graceful shutdown of the task worker. 2. Implementation: Within the task worker, the function logs the receipt of the SIGTERM signal and calls the

stop

method in the event loop thread to ensure a graceful shutdown. 3. Signal Registration: The signal handler is registered in the

start

method of the task worker. This is done using

_register_signal(signal.SIGTERM, self.handle_sigterm)

. Here's a conceptual snippet of how this might look in your code:

python
import signal
from prefect import task, flow

class TaskWorker:
    def handle_sigterm(self, signum, frame):
        <http://logger.info|logger.info>("SIGTERM received, initiating graceful shutdown...")
        from_sync.call_in_loop_thread(create_call(self.stop))
        sys.exit(0)

    @sync_compatible
    async def start(self):
        _register_signal(signal.SIGTERM, self.handle_sigterm)
        <http://logger.info|logger.info>("Starting task worker...")
        # Other startup logic...

# Example usage
worker = TaskWorker()
worker.start()

For more details, you can check the source code related to the task worker and signal handling. If you have any specific implementation details or questions, feel free to ask!

hi @Erik Forsman - can you explain what you ran into? we should handle this today

I'm using nomad to run workers. Upon a PR merge we startup a new container but any jobs running on an existing container get hung. I'm testing sending a SIGTERM now but that hasn't deployed yet and I'm wondering if any code changes are required to allow jobs to finish before the worker stops.