I'm asking Marvin at the same time <in the other c...
# ask-community
m
I'm asking Marvin at the same time in the other channel, but - has anybody experienced weird pauses between concurrency-limited tasks? This flow normally only takes 20-30min, but after update to the latest Prefect library, the flow is taking 4h+
m
Mine just hangs once I put any kind of concurrency. But maybe that's related to my other issue, because my task queue is in 10k tasks.
m
It seems ThreadPoolTaskRunner with its own limit of 20 tasks at a time works, while
rate_limit()
or
concurrency()
don't seem to do much on their own.
👍 1
Solid so far
Copy code
@flow(
    # ...
    task_runner=ThreadPoolTaskRunner(max_workers=20),