I have a running flow where a number of tasks are ...
# prefect-communitym
Matthew Seligson
07/07/2022, 7:18 PMI have a running flow where a number of tasks are in a paused state. I’d like to programmatically approve all of them. What’s the best way to do that? I find that when I programmatically set all of their states to resume, I get varying behavior. Sometimes the tasks stay resumed and never get running. Any ideas why?
k
Kevin Kho
07/07/2022, 7:29 PM
I don’t think there’s any other way other than the set task run state mutation which you seem to already be using. Does it only happen on a batch submission? There are no issues for one task?
m
Matthew Seligson
07/07/2022, 7:41 PMWe are setting them one at a time. The only other thing is that we are setting the flow state from Running() to Running(context={some data}). Is that launching a new pod when we do that?
k
Kevin Kho
07/07/2022, 8:18 PM
I don’t know off the top of my head. Definitely a case I haven’t seen before. Does the pod actually stay on during Paused? I need to try this out
m
Matthew Seligson
07/07/2022, 8:19 PMI would think the pod stays on ….
Matthew Seligson
07/07/2022, 8:20 PMHere’s what we’re doing
Matthew Seligson
07/07/2022, 8:21 PMWe first update the state of the flow to running, with state context hold=True. Then, all of our tasks have a state handler for when they transition to running. They get the flow’s state context and check for the hold=True. If they find it, raise a PAUSE signal.
Matthew Seligson
07/07/2022, 8:22 PMDoes that make sense so far? So a running flow will eventually have a bunch of tasks in a paused state, while the flow is still running. This essentially suspends execution.
Matthew Seligson
07/07/2022, 8:22 PMThen we want to resume execution, so we set all the paused tasks to resume. But like I mentioned, it doesn’t always work.
k
Kevin Kho
07/07/2022, 8:30 PM
Man this feels pretty complicated. What is the use case here? I don’t expect handling the context like this to work as expected (we don’t advice context manipulation). Wondering if you could use a polling type of setup instead
m
Matthew Seligson
07/07/2022, 8:37 PMFlows don’t have pause/resume states so we are trying to build that. We want to essentially pause a flow mid run without cancelling it and causing running tasks to fail. This is a functionality that our in house scheduler supported, and that we use frequently.
Also, this is not the normal context but the state context that gets serialized and store in Prefect cloud. I don’t think the problem is with the context part.
Matthew Seligson
07/07/2022, 8:40 PMThis is just a way of communicating to all tasks that they should raise a PAUSE signal. Now we need to reverse that.
k
Kevin Kho
07/07/2022, 9:17 PM
I am trying to test this. How do you pass a context in the first run and how do you resume it with a context? Do you have an example of that query?
m
Matthew Seligson
07/07/2022, 10:02 PMHaha I don’t have an example since that’s on my work comp
Matthew Seligson
07/07/2022, 10:02 PMI start the flow
Matthew Seligson
07/07/2022, 10:02 PMThen separately, from a jupyter notebook, I set the flow run state to Running(context=dict(on_hold=True))
Matthew Seligson
07/07/2022, 10:03 PMTo resume, I set the flow run state to Running() and then set all the paused tasks to resume
k
Kevin Kho
07/07/2022, 10:05 PM
Ok I see will try a bit tonight and see what I find. First thing is that the Docker container in DockerRun seems to go away
m
Matthew Seligson
07/08/2022, 3:02 PMWe are using kubernetes FYI
Matthew Seligson
07/08/2022, 3:02 PMAny luck?
k
Kevin Kho
07/08/2022, 3:10 PM
I ended up going out last night so didn’t get a chance. I can try more today, but I think you may get better help with the enterprise support on this one as they’ll be able to dig deeper. Similar to before, if you start a thread, I can chime in there also.
7 Views