Hey Yall. So we have The pattern of a Driver fl...
# prefect-communityr
Roger Webb
06/16/2022, 1:30 PMHey Yall. So we have The pattern of a Driver flow that calls child flows. Its working wonderfully where we have the create and the the Waitfor... however as the driver grows to control more flows... our waitfors are starting to error saying that the flow didnt start within 15 seconds. Once this errors... all the dependancy flows dont kick off, so its getting rather annoying. We are setting the number of workers... which we had at 10.. now lowering to 4 just to test it out... With unlimitted flow concurrency, shouldnt our flows be starting faster and not hitting that 15 second wall?
Roger Webb
06/16/2022, 1:48 PMJust as a point... running it with Num_workers=4 helped, but took a lot longer (as expected)
k
Kevin Kho
06/16/2022, 2:32 PM
Do you have traceback or logs I can see?
Kevin Kho
06/16/2022, 2:32 PM
I think if you run 10 at a time, the flows are synchronous so they will wait and occupy that thread before firing off the next. So you are limited by the Executor n_workers.
r
Roger Webb
06/16/2022, 2:43 PMWhen we didnt have that ... we had other issues with trying to insert into a table (snowflake) with more than 20 inserts at a time.. Is there a page on how to get the traceback?
k
Kevin Kho
06/16/2022, 2:45 PM
Ah I see. No I was just hoping for flow logs. Just to be sure you know, you could potentially limit the number of tasks accessing Snowflake.
Kevin Kho
06/16/2022, 2:45 PM
If it’s this error you get specifically, you could try the suggestion for threaded heartbeats. It might be more stable
r
Roger Webb
06/16/2022, 3:00 PMIts not the heartbeats... But the first looks at what I might have to do. THanks!
Roger Webb
06/16/2022, 4:06 PMSo Ive read through that doc.. and seems like thats what I need to do. Just one question for ya @Kevin Kho
Roger Webb
06/16/2022, 4:07 PMDoh.. forgot the question... lol.. It says to add the tag on the task. What would that be to limit the Creates and waitfor pairs? or would it be in the child flows..?
Roger Webb
06/16/2022, 4:12 PMWont that make the waitfors fail more if the parent calls the create and the create doesnt happen until a thread is ready?
k
Kevin Kho
06/16/2022, 5:05 PM
It would be in the child flows that access the Snowflake task. This way, you can just trigger more child flows without necessarily caring right?
r
Roger Webb
06/16/2022, 5:05 PMI can see where that might work.. 🙂 Let me give it a whirl.
5 Views