https://prefect.io logo
Title
j

Jeff Brainerd

08/14/2020, 7:39 PM
Hey šŸ˜›refect: team, a question about dask interaction. We are looking for best practices for interrupting and then restarting a flow. Our use case is that we need to sever all database connections during a database migration. We would like to avoid restarting the flow from the beginning, but we are fine with re-starting any running tasks from the beginning (in fact I don’t think we have a choice there). We are thinking of the following steps: • pause all running tasks using graphql API • scale down dask workers to 0 (we run in fargate) • scale up dask workers after the migration • unpause the tasks Would love to hear any thoughts. Thanks! šŸ™
šŸ‘€ 3
c

Chris White

08/14/2020, 9:16 PM
Hey Jeff - based on what you describe, here’s what I’d recommend: - ā€œcancelā€ the Flow using the new cancel button in the UI (assuming you are running on a newer version of Core) - once the Flow is fully cancelled, you’ll have a bunch of task runs in a ā€œCancelledā€ state - you might need to use the GraphQL API to set all ā€œCancelledā€ task runs into a ā€œPendingā€ state - finally, Mark your flow run as ā€œScheduledā€ and it will get picked up and executed from the pending task run states We could probably work on making this experience more intuitive so you don’t have to jump around so much
:upvote: 1
Also one other thing to note: if you place your flow run into any
Finished
state (Success, Failed, Finished), no new task runs will begin their run and all currently running will complete gracefully. Then you could restart the flow without having to deal with the graphql API at the cost of waiting for currently running tasks to complete
j

Jeff Brainerd

08/15/2020, 11:11 PM
Thanks for getting back so quickly @Chris White! At least for now I will go with the first option, because we don’t want to hold up a deploy to wait for a potentially long running task to complete. BTW we will do this all via the graphql API because we want the whole thing to run without human intervention as part of the deployment process.