I’ve got a backfilling flow that is supposed to set off several thousand tasks, which ideally I’d have on the ConcurrentTaskRunner. When running locally on a powerful laptop this works fine, but on an Agent its CPU goes right to its limit but only a few tasks actually complete before it just stops doing anything. Is this to be expected at the minute or a surprise? Would this go better if I broke it up into lots of subflows instead of lots of tasks?
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.