https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> Is there a way to tell a worker (proc...
# ask-marvin
t
@Marvin Is there a way to tell a worker (process worker or whatever) to "stop after you finish running this flow"? similar to 
--run-once
m
✅ thought for 61.9 seconds
Yes — in Prefect 3.x, workers support 
--run-once
. Example:
Copy code
bash
prefect worker start --pool my-pool --type process --run-once
That tells the worker to: 1. poll for work once 2. start any flow runs it picks up 3. wait for those active runs to finish 4. exit So it’s effectively “run one polling cycle, then shut down once the work you claimed is done.” A couple notes: - It is not “stop after the currently running flow on an already-running worker” - It is “start this worker in one-shot mode” - This works for process workers too CLI help confirms the option exists: 
prefect worker start --help
Relevant docs: - Workers - Worker CLI reference If what you mean is slightly different — e.g. signal an already-running worker to drain and exit after its current flow run — that’s a different behavior, and Prefect doesn’t expose a simple “drain after current job” flag on a live worker the same way some queue systems do. In that case I can suggest a few patterns depending on whether you’re using process, docker, or k8s workers.
t
@Marvin No, not --run-once, I have a process worker that has been running for a while, and I want it to stop after this current flow is done.
m
✅ thought for 206.6 seconds
n
no there isn't
🥀 1
what's the use case?
is the fact that the worker has been running a while related in some way to the reason you want to stop it? bc typically workers run forever
t
@Nate I'm using pre-emptible pods on k8s for some tasks just to saturate compute, they got terminated easily and I have a 5 minute backoff, so I want to tell my prefect worker to wrap this up and don't take any new jobs after this. Currently i am doing aggresive flow-run heartbeat + prune zombie flows but I feel like there has to be a better approach. Plus heartbeat does not work when the jobs got stuck on PENDING afaik
m
Got it — for an already-running process worker, there isn’t currently a built-in “drain and exit after the current flow run finishes” command/flag. So the short answer is: not directly. What you can do: Option 1: pause the work pool / work queue, then stop the worker yourself once it goes idle - This prevents it from picking up new runs - But it does not make the worker auto-exit after the current run finishes Option 2: send SIGTERM / stop the worker process - This stops the worker - But for a normal long-running worker, this is not the same as a graceful “wait for active flow runs to finish, then exit” - The special “wait for active runs before exiting” behavior is tied to 
--run-once
So if your goal is exactly:
“don’t take any more work, but let the current run finish, then exit”
Prefect workers don’t currently expose that as a first-class drain mode. A practical workaround for a single worker is: 1. make sure it won’t claim new work - pause the work pool, or - restrict/remove the queue it polls, or - otherwise isolate it from new assignments 2. wait until its current flow run completes 3. terminate the worker process If you want, I can help you pick the safest approach based on how this worker is targeted: - dedicated work pool? - specific work queue? - multiple workers sharing the same pool? - process launched via systemd / docker / kubernetes / plain shell? Relevant docs: - Workers - Worker CLI
t
feel like it's a better approach that what I am currently doing (e.g. heartbeat + prune zombie flows)
n
interesting. can you elaborate on why you're using a process worker in a pod? if that's a correct interpretation? is it startup time?
like using a k8s work pool and pausing the pool seems natural naively
t
@Nate well our org uses Nvidia RunAI which wraps around k8s for GPU timeshare purpose, so the cluster is shared amount different teams + we don't have full control on the k8s level. Using a process worker/workpool + a customized container with parallelism is the best I come up with. I never really dig deeper into k8s workflow so I could be missing out on it lol
so I have my Dockerfile as:
Copy code
# Entrypoint: starts Prefect worker; names it after the pod by default
ENTRYPOINT ["/usr/bin/tini", "-g", "--", "/usr/local/bin/prefect-entrypoint.sh"]
then in the script
Copy code
exec /opt/venv/bin/python -m \
  prefect worker start \
  --pool "$POOL" \
  --limit "$LIMIT" \
  --name "$WORKER_NAME" \
  "$@"
n
thanks for the context!
we don't have full control on the k8s level
can you create jobs with the access that you have?
t
yes we can but it's thru runai CLI which handles all the k8s level pod creation
kubectl is view only
2 Views
Open in Slack
PreviousNext
Powered by