Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Can I set an upper bound for tasks run in parallel using `apply_map` ?

My problem is, that when I map more than 60 Tasks at once, each having an open connection to an external database, the flow will remain in a running state and never terminate.

Actually, this already occurs for 30 parallel Tasks.
As background: I have a flow that connects to a database to retrieve  2minute statistics, e.g.
``` [2020-08-08 10:00:00, 2020-08-08 10:02:00), [2020-08-08 10:02:00, 2020-08-08 10:04:00).. ```
over the last hour and writes it to another database.

Hi <@U016L5DJTED>! You might want to take a look at Task Concurrency Limiting: <https://docs.prefect.io/orchestration/concepts/task-concurrency-limiting.html#task-concurrency-limiting>