https://prefect.io logo
Title
l

Lawrence Finn

06/23/2021, 7:48 PM
Is there a safe way to stop an agent? I am worried about stopping it mid DAG run
k

Kevin Kho

06/23/2021, 7:52 PM
Hey, why are you trying to stop the agent mid run? The agent is a pretty lightweight process.
l

Lawrence Finn

06/23/2021, 7:56 PM
if i had to upgrade the machine or something? 🤷
is there a way to make it stop after it is done running what it is currently running?
if i understand correctly, it looks like if the agent stops heartbeating the dag will be marked failed
k

Kevin Kho

06/23/2021, 8:01 PM
The answer here is it depends. For something where the Flow is being performed on another machine like a Dask cluster, those processes are hard to kill so those can keep going. If the process is something running locally, it would be ended
I think to upgrade, you would spin another agent up, set labels on the old one such that it doesn’t pick up any new flows, then spin down when those running flows complete. No to the making it stop after finishing flow runs.
c

Chris White

06/23/2021, 8:12 PM
FYI The only agent that maintains state relevant for flow runs is the LocalAgent -- any other agent is safe to start / stop without it affecting any outstanding runs
l

Lawrence Finn

06/23/2021, 9:23 PM
@Chris White im new to prefect, what does that mean exactly 😄 If i start a kubernetes agent, it can die at any time and have no side effects?
c

Chris White

06/23/2021, 10:11 PM
Yup, that's essentially correct! The kubernetes agent does the following two things: • submits flow runs as a k8s job • tracks the jobs it submits to clean them up when they are complete So when you restart or turn off the agent, you do miss out on the pod cleanup step but in practice this shouldn't have a material impact on your workloads
l

Lawrence Finn

06/23/2021, 10:41 PM
oooo i was using a local agent like a chump. so how does it route the output of task A to task B (if i create that dependency). do the k8s tasks it creates handle that?
c

Chris White

06/23/2021, 10:43 PM
Haha no worries, the local agent is fully supported but is much less robust; the k8s job encapsulates the full flow run, so each individual task will run within that same job
l

Lawrence Finn

06/23/2021, 10:44 PM
OH
so the local one forks a process
the kubernetes one does a new pod per run?
c

Chris White

06/23/2021, 10:46 PM
yup, exactly
l

Lawrence Finn

06/23/2021, 10:47 PM
but then i guess a single “routing” agent runs in k8s to create pods for runs? or does that happen from the prefect server?