<@ULVA73B9P> Is it possible to ask a process worke...
# ask-marvin
t
@Marvin Is it possible to ask a process worker in a worker pool to shut down after finish running it's current job?
m
thought for 229.3 seconds
Yes. In Prefect 3.x you have two good options: 1) Start the worker in “run once” mode (will pick up at most one flow run then exit) - CLI:
Copy code
prefect worker start -p <your-pool> -n <optional-name> --run-once
- Python:
Copy code
from prefect.workers.process import ProcessWorker

worker = ProcessWorker(work_pool_name="<your-pool>", work_queues=["<optional-queue>"])
worker.start(run_once=True)
Docs: Process worker API 2) Gracefully stop an already-running worker after its current run(s) - Pause the pool so the worker won’t fetch new work:
Copy code
prefect work-pool pause <your-pool>
- Then send a SIGTERM to the worker process (the worker will stop polling and finish any in-flight runs before exiting): - Linux/macOS:
kill -TERM <worker_pid>
- Docker:
docker stop <container>
- Kubernetes: scale to 0 or delete the pod (ensure a sufficient
terminationGracePeriodSeconds
) Implementation note: workers handle SIGTERM gracefully and will shut down cleanly after completing active work. Reference: task_worker.py If you can share how you’re running the worker (local process, Docker, Kubernetes), I can suggest the exact commands/configuration.
t
@Marvin Just to confirm, sending a SIGTERM to that process would not make it terminate it
m
thought for 20.5 seconds
Short answer: correct—sending SIGTERM to the Prefect 3 process worker triggers a graceful shutdown. It will stop polling for new work and, by default, let any in-flight flow run(s) finish before the worker exits. It does not immediately kill the current job. A few caveats: - If you send SIGTERM to the flow-run process itself (not the worker), that will terminate the run. - In Docker/Kubernetes, make sure the signal reaches the worker process and that your termination grace period is long enough; otherwise the platform may escalate to SIGKILL. - If you want to guarantee no new work is picked up before shutdown, pause the pool first, then send SIGTERM. Or start the worker with
--run-once
. Refs: - Process worker API: prefect.workers.process - Worker/task worker shutdown handling: task_worker.py If you can share your runtime (local, Docker, Kubernetes), I can recommend the safest shutdown pattern for that environment.
t
@Marvin It's Nvidia RunAI, which is k8s under the hood
m
thought for 71.9 seconds
Got it — for RunAI (Kubernetes): What happens on SIGTERM - Deleting the Pod or scaling a Deployment to 0 sends SIGTERM to the worker process. - The Prefect 3 process worker handles SIGTERM gracefully: it stops polling, lets any in‑flight flow run(s) finish, then exits. It does not kill the current run. - Ensure your Pod’s
terminationGracePeriodSeconds
is long enough to let the active run(s) complete; otherwise Kubernetes will escalate to SIGKILL and terminate the run. Two safe patterns to shut down after the current job 1) Ephemeral “run once” worker (recommended for K8s/RunAI) - Start the worker with
--run-once
so it picks up at most one flow run and then exits. In K8s, run it as a Job (it will complete when the worker exits).
Copy code
prefect worker start -p <pool> --run-once --limit 1
- Using a Job per worker gives you clean, automatic teardown. 2) Gracefully stop a long‑running worker - Pause the pool to prevent new work:
Copy code
prefect work-pool pause <pool>
- Then delete the Pod (or scale the Deployment to 0). K8s sends SIGTERM; the worker will finish current run(s) and exit before the grace period ends. Kubernetes tips - Set
terminationGracePeriodSeconds
high enough for your longest run. Example: 3600 (1 hour). - If you have multiple concurrent runs per worker, consider
--limit 1
to ensure only one in-flight run at shutdown. - Ensure the signal reaches the worker process (PID 1 inside the container). References - Process worker API: prefect.workers.process - Work pool pause CLI help (verified):
prefect work-pool pause --help
If you share how you’re launching the worker in RunAI (Deployment vs. Job, command/args), I can suggest a concrete manifest/command snippet.
t
Copy code
ps aux | grep prefect
root           1  0.0  0.0 142304  5800 ?        Ssl  Sep19   0:00 uv run --python /opt/venv/bin/python prefect worker start --pool runai-test --limit 1 --name tom-prefect-worker-test-3-1-2
root           9  0.3  0.0 3227816 225808 ?      Sl   Sep19  56:51 /opt/venv/bin/python /opt/venv/bin/prefect worker start --pool runai-test --limit 1 --name tom-prefect-worker-test-3-1-2
root       11018 1648  1.3 20502252 7041136 ?    Rl   09:44 12667:27 /opt/venv/bin/python -m prefect.engine
root       11219  0.0  0.0   5104  2048 pts/0    S+   22:32   0:00 grep --color=auto prefect
@Marvin which process should I send the sig kill to?
m
thought for 32.8 seconds
Thanks for the
ps
output — here’s what you’ve got: - PID 1:
uv run ... prefect worker start ...
(launcher) - PID 9:
/opt/venv/bin/python /opt/venv/bin/prefect worker start ...
(the Prefect worker process) - PID 11018:
/opt/venv/bin/python -m prefect.engine
(the flow-run process currently executing your job) To gracefully shut down the worker after it finishes the current job, send SIGTERM to the worker process — PID 9. Do not signal PID 11018 (that would kill the current run), and avoid signaling PID 1 (the uv launcher) since it may not forward signals reliably. Example:
Copy code
kubectl exec <pod> -- kill -TERM 9
Optional safety step to prevent picking up new work:
Copy code
prefect work-pool pause runai-test
If you want this behavior by default in the future, start the worker with
--run-once
so it exits after handling one run:
Copy code
prefect worker start -p runai-test --limit 1 --run-once
Refs: - Process worker API: prefect.workers.process
t
@Marvin How do I unpause a worker pool?
m
thought for 13.4 seconds
Use:
Copy code
prefect work-pool resume <your-pool-name>
Example:
Copy code
prefect work-pool resume runai-test
You can verify the state with:
Copy code
prefect work-pool inspect <your-pool-name>
(help checked:
prefect work-pool resume --help
)