https://prefect.io
Join Slack
Maybe a small question, but I cannot find the answer in the docs. Is it possible to use Prefect to p...
# ask-community
m
Maybe a small question, but I cannot find the answer in the docs. Is it possible to use Prefect to process minibatches in which a flow is scheduled to run at, let's say, every 5 mins but can only run in case the previous run is completed (can be successful or failed)? Thanks in advance for the info!
z
you can use tags to set flow concurrency, so itll only ever run one instance of that flow at a time
believe you have to be on cloud, tho
k
Hey @Matthias Roels, flow concurrency may help here but it would not cancel the extra flow run, just limit the number of concurrent runs. I think what you need is to use a task that hits the GraphQL API to query and see if that no other flow is running. If no other flow is running, then proceed with the rest of your flow.
Query to get you started:
Copy code
query {
  flow_run(
    where: {flow: {name: {_eq: "flow_name"}}}
    order_by: {created: desc}
  ) {
    name
    id
    state
	flow {
      name
    }
  }
}
Then check the state if it’s SUCCESS or FAILED. If it’s not either, raise the 
ENDRUN
signal
m
Alright! Thanks for the feedback!
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by