Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

image.png

Hi :wave: ! First of all, thank you for such a great tool, I really enjoy using prefect :slightly_smiling_face: I'm using prefect-ui with hourly schedule to run an ETL flow. The agent is in a raspberrypi4 :see_no_evil: , and it sometimes get discconected from the network (Maybe I should stop using a Raspberry for this, I know). When this happen I see that all the missing hourly flows start executing when the agent is turned on (all at the same time). Is there any way to configure prefect to not start executing flows *after* the schedule time? Should I do it programatticaly? Or maybe just don't allow it to execute the same flow at the same time.

Here is an example where many flows started running together:

Hi <@U01RXPPK9KP>, in Standard tier and up we offer Flow automations where you can say “don’t run a flow if it’s late by X amount”. What are you using the raspberry pi for? I’m curious

Nice to know ! I'm interested in this feature so I'll check the standard tier for sure.

About the raspberry: Basically I'm new at a startup (in Argentina) and I still havent had time to learn how to deploy in the company's cloud instances, so since my ETLs use the compute power of a Data Warehouse I don't need too much compute in the agent host, so I decided to use a raspberry that was at hand to run the schedules. :stuck_out_tongue:

The problem is when the raspberry get's discconected from the internet, ISP's are not so stable here.

Thanks for your help, <@U01QEJ9PP53>!

About executing the same flow twice concurrently at the same time, is there some way to avoid this?

Is it because the schedule got delayed? Or is there multiple schedules/run creations and you just want something to block 2 of the same flow from running concurrently?

In any case where there are mulitple executions of the same flow I would like to be able to block all except the one who started first.

I’m starting to sound like I’m selling aggressively, but limiting flow run concurrency is also on the standard tier :sweat_smile:

hahaha, thats ok ! it's nice to know it again, I will better go and check the tier features :stuck_out_tongue:

Thanks a lot for your time <@U01QEJ9PP53>, nice to meet you :slightly_smiling_face: