https://prefect.io logo
Title
m

Matthew Seligson

07/07/2022, 7:18 PM
I have a running flow where a number of tasks are in a paused state. I’d like to programmatically approve all of them. What’s the best way to do that? I find that when I programmatically set all of their states to resume, I get varying behavior. Sometimes the tasks stay resumed and never get running. Any ideas why?
k

Kevin Kho

07/07/2022, 7:29 PM
I don’t think there’s any other way other than the set task run state mutation which you seem to already be using. Does it only happen on a batch submission? There are no issues for one task?
m

Matthew Seligson

07/07/2022, 7:41 PM
We are setting them one at a time. The only other thing is that we are setting the flow state from Running() to Running(context={some data}). Is that launching a new pod when we do that?
k

Kevin Kho

07/07/2022, 8:18 PM
I don’t know off the top of my head. Definitely a case I haven’t seen before. Does the pod actually stay on during Paused? I need to try this out
m

Matthew Seligson

07/07/2022, 8:19 PM
I would think the pod stays on ….
Here’s what we’re doing
We first update the state of the flow to running, with state context hold=True. Then, all of our tasks have a state handler for when they transition to running. They get the flow’s state context and check for the hold=True. If they find it, raise a PAUSE signal.
Does that make sense so far? So a running flow will eventually have a bunch of tasks in a paused state, while the flow is still running. This essentially suspends execution.
Then we want to resume execution, so we set all the paused tasks to resume. But like I mentioned, it doesn’t always work.
k

Kevin Kho

07/07/2022, 8:30 PM
Man this feels pretty complicated. What is the use case here? I don’t expect handling the context like this to work as expected (we don’t advice context manipulation). Wondering if you could use a polling type of setup instead
m

Matthew Seligson

07/07/2022, 8:37 PM
Flows don’t have pause/resume states so we are trying to build that. We want to essentially pause a flow mid run without cancelling it and causing running tasks to fail. This is a functionality that our in house scheduler supported, and that we use frequently. Also, this is not the normal context but the state context that gets serialized and store in Prefect cloud. I don’t think the problem is with the context part.
This is just a way of communicating to all tasks that they should raise a PAUSE signal. Now we need to reverse that.
k

Kevin Kho

07/07/2022, 9:17 PM
I am trying to test this. How do you pass a context in the first run and how do you resume it with a context? Do you have an example of that query?
m

Matthew Seligson

07/07/2022, 10:02 PM
Haha I don’t have an example since that’s on my work comp
I start the flow
Then separately, from a jupyter notebook, I set the flow run state to Running(context=dict(on_hold=True))
To resume, I set the flow run state to Running() and then set all the paused tasks to resume
k

Kevin Kho

07/07/2022, 10:05 PM
Ok I see will try a bit tonight and see what I find. First thing is that the Docker container in DockerRun seems to go away
m

Matthew Seligson

07/08/2022, 3:02 PM
We are using kubernetes FYI
Any luck?
k

Kevin Kho

07/08/2022, 3:10 PM
I ended up going out last night so didn’t get a chance. I can try more today, but I think you may get better help with the enterprise support on this one as they’ll be able to dig deeper. Similar to before, if you start a thread, I can chime in there also.