https://prefect.io logo
Title
e

Eli Treuherz

09/08/2022, 2:04 PM
I’ve got a backfilling flow that is supposed to set off several thousand tasks, which ideally I’d have on the ConcurrentTaskRunner. When running locally on a powerful laptop this works fine, but on an Agent its CPU goes right to its limit but only a few tasks actually complete before it just stops doing anything. Is this to be expected at the minute or a surprise? Would this go better if I broke it up into lots of subflows instead of lots of tasks?
r

Rob Freedy

09/09/2022, 3:14 PM
Hey Eli!! There are a couple options in general for solving this type of issue. I would recommend taking a look at this discourse post as it has some good suggestions for handling CPU/GPU for high volume flows: https://discourse.prefect.io/t/how-to-handle-custom-resource-cpu-vs-gpu-and-code-dependencies-across-various-tasks-and-flows/503