Hey Yall So we have The pattern of a Driver flow that calls

Question

Hey Yall So we have The pattern of a Driver flow that calls child flows Its working wonderfully where we have the create and the the Waitfor however as the driver grows to control more flows our waitfors are starting to error saying that the flow didnt start within 15 seconds Once this errors all the dependancy flows dont kick off so its getting rather annoying We are setting the number of workers which we had at 10 now lowering to 4 just to test it out With unlimitted flow concurrency shouldnt our flows be starting faster and not hitting that 15 second wall

Roger Webb · Answer

Just as a point running it with Num workers=4 helped but took a lot longer as expected

Kevin Kho · Answer

Do you have traceback or logs I can see

Kevin Kho · Answer

I think if you run 10 at a time the flows are synchronous so they will wait and occupy that thread before firing off the next So you are limited by the Executor n workers

Roger Webb · Answer

When we didnt have that we had other issues with trying to insert into a table snowflake with more than 20 inserts at a time Is there a page on how to get the traceback

Kevin Kho · Answer

Ah I see No I was just hoping for flow logs Just to be sure you know you could potentially <https docs prefect io orchestration flow runs concurrency limits html task run limits|limit the number of tasks> accessing Snowflake

Kevin Kho · Answer

If it s <https docs prefect io orchestration faq faq html why does prefect mark flow runs with no heartbeat detected from the remote task marking the run as failed|this> error you get specifically you could try the suggestion for threaded heartbeats It might be more stable

Roger Webb · Answer

Its not the heartbeats But the first looks at what I might have to do THanks

Roger Webb · Answer

So Ive read through that doc and seems like thats what I need to do Just one question for ya < Kevin Kho>

Roger Webb · Answer

Doh forgot the question lol It says to add the tag on the task What would that be to limit the Creates and waitfor pairs or would it be in the child flows

Roger Webb · Answer

Wont that make the waitfors fail more if the parent calls the create and the create doesnt happen until a thread is ready

Kevin Kho · Answer

It would be in the child flows that access the Snowflake task This way you can just trigger more child flows without necessarily caring right

Roger Webb · Answer

I can see where that might work slightly smiling face Let me give it a whirl