Tom Han
09/29/2025, 9:40 PMMarvin
09/29/2025, 9:40 PMMarvin
09/29/2025, 9:44 PMprefect worker start -p <your-pool> -n <optional-name> --run-once
- Python:
from prefect.workers.process import ProcessWorker
worker = ProcessWorker(work_pool_name="<your-pool>", work_queues=["<optional-queue>"])
worker.start(run_once=True)
Docs: Process worker API
2) Gracefully stop an already-running worker after its current run(s)
- Pause the pool so the worker won’t fetch new work:
prefect work-pool pause <your-pool>
- Then send a SIGTERM to the worker process (the worker will stop polling and finish any in-flight runs before exiting):
- Linux/macOS: kill -TERM <worker_pid>
- Docker: docker stop <container>
- Kubernetes: scale to 0 or delete the pod (ensure a sufficient terminationGracePeriodSeconds
)
Implementation note: workers handle SIGTERM gracefully and will shut down cleanly after completing active work. Reference: task_worker.py
If you can share how you’re running the worker (local process, Docker, Kubernetes), I can suggest the exact commands/configuration.Tom Han
09/29/2025, 9:46 PMMarvin
09/29/2025, 9:46 PMMarvin
09/29/2025, 9:46 PM--run-once
.
Refs:
- Process worker API: prefect.workers.process
- Worker/task worker shutdown handling: task_worker.py
If you can share your runtime (local, Docker, Kubernetes), I can recommend the safest shutdown pattern for that environment.Tom Han
09/29/2025, 10:09 PMMarvin
09/29/2025, 10:09 PMMarvin
09/29/2025, 10:11 PMterminationGracePeriodSeconds
is long enough to let the active run(s) complete; otherwise Kubernetes will escalate to SIGKILL and terminate the run.
Two safe patterns to shut down after the current job
1) Ephemeral “run once” worker (recommended for K8s/RunAI)
- Start the worker with --run-once
so it picks up at most one flow run and then exits. In K8s, run it as a Job (it will complete when the worker exits).
prefect worker start -p <pool> --run-once --limit 1
- Using a Job per worker gives you clean, automatic teardown.
2) Gracefully stop a long‑running worker
- Pause the pool to prevent new work:
prefect work-pool pause <pool>
- Then delete the Pod (or scale the Deployment to 0). K8s sends SIGTERM; the worker will finish current run(s) and exit before the grace period ends.
Kubernetes tips
- Set terminationGracePeriodSeconds
high enough for your longest run. Example: 3600 (1 hour).
- If you have multiple concurrent runs per worker, consider --limit 1
to ensure only one in-flight run at shutdown.
- Ensure the signal reaches the worker process (PID 1 inside the container).
References
- Process worker API: prefect.workers.process
- Work pool pause CLI help (verified): prefect work-pool pause --help
If you share how you’re launching the worker in RunAI (Deployment vs. Job, command/args), I can suggest a concrete manifest/command snippet.Tom Han
09/29/2025, 10:33 PMps aux | grep prefect
root 1 0.0 0.0 142304 5800 ? Ssl Sep19 0:00 uv run --python /opt/venv/bin/python prefect worker start --pool runai-test --limit 1 --name tom-prefect-worker-test-3-1-2
root 9 0.3 0.0 3227816 225808 ? Sl Sep19 56:51 /opt/venv/bin/python /opt/venv/bin/prefect worker start --pool runai-test --limit 1 --name tom-prefect-worker-test-3-1-2
root 11018 1648 1.3 20502252 7041136 ? Rl 09:44 12667:27 /opt/venv/bin/python -m prefect.engine
root 11219 0.0 0.0 5104 2048 pts/0 S+ 22:32 0:00 grep --color=auto prefect
@Marvin which process should I send the sig kill to?Marvin
09/29/2025, 10:33 PMMarvin
09/29/2025, 10:34 PMps
output — here’s what you’ve got:
- PID 1: uv run ... prefect worker start ...
(launcher)
- PID 9: /opt/venv/bin/python /opt/venv/bin/prefect worker start ...
(the Prefect worker process)
- PID 11018: /opt/venv/bin/python -m prefect.engine
(the flow-run process currently executing your job)
To gracefully shut down the worker after it finishes the current job, send SIGTERM to the worker process — PID 9. Do not signal PID 11018 (that would kill the current run), and avoid signaling PID 1 (the uv launcher) since it may not forward signals reliably.
Example:
kubectl exec <pod> -- kill -TERM 9
Optional safety step to prevent picking up new work:
prefect work-pool pause runai-test
If you want this behavior by default in the future, start the worker with --run-once
so it exits after handling one run:
prefect worker start -p runai-test --limit 1 --run-once
Refs:
- Process worker API: prefect.workers.processTom Han
09/29/2025, 11:15 PMMarvin
09/29/2025, 11:15 PMMarvin
09/29/2025, 11:16 PMprefect work-pool resume <your-pool-name>
Example:
prefect work-pool resume runai-test
You can verify the state with:
prefect work-pool inspect <your-pool-name>
(help checked: prefect work-pool resume --help
)