Eli Treuherz

09/08/2022, 2:04 PM
I’ve got a backfilling flow that is supposed to set off several thousand tasks, which ideally I’d have on the ConcurrentTaskRunner. When running locally on a powerful laptop this works fine, but on an Agent its CPU goes right to its limit but only a few tasks actually complete before it just stops doing anything. Is this to be expected at the minute or a surprise? Would this go better if I broke it up into lots of subflows instead of lots of tasks?

Rob Freedy

09/09/2022, 3:14 PM
Hey Eli!! There are a couple options in general for solving this type of issue. I would recommend taking a look at this discourse post as it has some good suggestions for handling CPU/GPU for high volume flows: