Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
m
Matthew Seligson
07/07/2022, 7:18 PMI have a running flow where a number of tasks are in a paused state. I’d like to programmatically approve all of them. What’s the best way to do that? I find that when I programmatically set all of their states to resume, I get varying behavior. Sometimes the tasks stay resumed and never get running. Any ideas why?
k
Kevin Kho
07/07/2022, 7:29 PMI don’t think there’s any other way other than the set task run state mutation which you seem to already be using. Does it only happen on a batch submission? There are no issues for one task?
m
Matthew Seligson
07/07/2022, 7:41 PMWe are setting them one at a time. The only other thing is that we are setting the flow state from Running() to Running(context={some data}). Is that launching a new pod when we do that?
k
Kevin Kho
07/07/2022, 8:18 PMI don’t know off the top of my head. Definitely a case I haven’t seen before. Does the pod actually stay on during Paused? I need to try this out
m
Matthew Seligson
07/07/2022, 8:19 PMI would think the pod stays on ….
Here’s what we’re doing
We first update the state of the flow to running, with state context hold=True. Then, all of our tasks have a state handler for when they transition to running. They get the flow’s state context and check for the hold=True. If they find it, raise a PAUSE signal.
Does that make sense so far? So a running flow will eventually have a bunch of tasks in a paused state, while the flow is still running. This essentially suspends execution.
Then we want to resume execution, so we set all the paused tasks to resume. But like I mentioned, it doesn’t always work.
k
Kevin Kho
07/07/2022, 8:30 PMMan this feels pretty complicated. What is the use case here? I don’t expect handling the context like this to work as expected (we don’t advice context manipulation). Wondering if you could use a polling type of setup instead
m
Matthew Seligson
07/07/2022, 8:37 PMFlows don’t have pause/resume states so we are trying to build that. We want to essentially pause a flow mid run without cancelling it and causing running tasks to fail. This is a functionality that our in house scheduler supported, and that we use frequently.
Also, this is not the normal context but the state context that gets serialized and store in Prefect cloud. I don’t think the problem is with the context part.
This is just a way of communicating to all tasks that they should raise a PAUSE signal. Now we need to reverse that.
k
Kevin Kho
07/07/2022, 9:17 PMI am trying to test this. How do you pass a context in the first run and how do you resume it with a context? Do you have an example of that query?
m
Matthew Seligson
07/07/2022, 10:02 PMHaha I don’t have an example since that’s on my work comp
I start the flow
Then separately, from a jupyter notebook, I set the flow run state to Running(context=dict(on_hold=True))
To resume, I set the flow run state to Running() and then set all the paused tasks to resume
k
Kevin Kho
07/07/2022, 10:05 PMOk I see will try a bit tonight and see what I find. First thing is that the Docker container in DockerRun seems to go away
m
Matthew Seligson
07/08/2022, 3:02 PMWe are using kubernetes FYI
Any luck?
k
Kevin Kho
07/08/2022, 3:10 PMI ended up going out last night so didn’t get a chance. I can try more today, but I think you may get better help with the enterprise support on this one as they’ll be able to dig deeper. Similar to before, if you start a thread, I can chime in there also.