Hey Yall. So we have The pattern of a Driver flow that calls child flows. Its working wonderfully where we have the create and the the Waitfor... however as the driver grows to control more flows... our waitfors are starting to error saying that the flow didnt start within 15 seconds. Once this errors... all the dependancy flows dont kick off, so its getting rather annoying. We are setting the number of workers... which we had at 10.. now lowering to 4 just to test it out... With unlimitted flow concurrency, shouldnt our flows be starting faster and not hitting that 15 second wall?
Just as a point... running it with Num_workers=4 helped, but took a lot longer (as expected)
06/16/2022, 2:32 PM
Do you have traceback or logs I can see?
I think if you run 10 at a time, the flows are synchronous so they will wait and occupy that thread before firing off the next. So you are limited by the Executor n_workers.
06/16/2022, 2:43 PM
When we didnt have that ... we had other issues with trying to insert into a table (snowflake) with more than 20 inserts at a time.. Is there a page on how to get the traceback?
06/16/2022, 2:45 PM
Ah I see. No I was just hoping for flow logs. Just to be sure you know, you could potentially limit the number of tasks accessing Snowflake.
If it’s this error you get specifically, you could try the suggestion for threaded heartbeats. It might be more stable
06/16/2022, 3:00 PM
Its not the heartbeats... But the first looks at what I might have to do. THanks!
So Ive read through that doc.. and seems like thats what I need to do. Just one question for ya @Kevin Kho
Doh.. forgot the question... lol.. It says to add the tag on the task. What would that be to limit the Creates and waitfor pairs? or would it be in the child flows..?
Wont that make the waitfors fail more if the parent calls the create and the create doesnt happen until a thread is ready?
06/16/2022, 5:05 PM
It would be in the child flows that access the Snowflake task. This way, you can just trigger more child flows without necessarily caring right?
06/16/2022, 5:05 PM
I can see where that might work.. 🙂 Let me give it a whirl.