I’ve got a backfilling flow that is supposed to se...
# prefect-community
I’ve got a backfilling flow that is supposed to set off several thousand tasks, which ideally I’d have on the ConcurrentTaskRunner. When running locally on a powerful laptop this works fine, but on an Agent its CPU goes right to its limit but only a few tasks actually complete before it just stops doing anything. Is this to be expected at the minute or a surprise? Would this go better if I broke it up into lots of subflows instead of lots of tasks?
Hey Eli!! There are a couple options in general for solving this type of issue. I would recommend taking a look at this discourse post as it has some good suggestions for handling CPU/GPU for high volume flows: https://discourse.prefect.io/t/how-to-handle-custom-resource-cpu-vs-gpu-and-code-dependencies-across-various-tasks-and-flows/503