https://prefect.io
Join Slack
Hi, is there a way to make schedule to run one after one? like, if the first job didn’t finish, the ...
# ask-community
t
Hi, is there a way to make schedule to run one after one? like, if the first job didn’t finish, the second one has to wait.
k
Hey @Tony Yun, are these for the same flow?
t
yes
k
Maybe you can use flow run concurrency ?
t
ah I see. Sounds good to me. Thanks!
hold on, is there a way to trigger the job to run after it finishes a certain amount of time? that not on a fixed schedule
k
It would have to be through the state handler of the currently running flow to kick of a new flow run through 
create_flow_run
or 
StartFlowRun
. What is the use case you are trying to achieve? Maybe it can be done with automations also
t
we have a dbt job that is time consuming and can’t run in parallel. however the time spent is inconsistent so can’t use fixed schedule either..
k
What do you want to run after the dbt job?
t
the same dbt job
like to run it repeatedily
k
Ah in this case, you want to just have a flow level state handler that creates a new flow run upon completion
upvote 1
Hey I take that back, what you should do is use flow concurrency of 1, and then if the flow run is too late, use an automation to cancel it
👍 1
so that it will just continue from the next next schedule you know?
I think an ELT architecture might help you more than ETL btw if you need to be running dbt constantly
This article by the Airbyte CEO is pretty good
t
Thank you for this much information! I’ll take a look around.
6 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by