suppose I have a mapped task that connects to a certain DB. I’d like it to run parallel, but I’d also like to limit the number of concurrent running copies of the task, regardless of the number of available executors in my cluster (e.g. I might have 32 executors but the DB can handle only 4 connections). Is there a way to do that in prefect?
05/21/2020, 1:05 PM
With Prefect Cloud, we have a global concurrency lock for tasks with a certain tag (across all flows on a team) for this exact purpose