08/09/2023, 11:24 AM
@Marvin is it possible to use a Lock in a task to avoid race conditions?


08/09/2023, 11:25 AM
Oh, the joy of concurrency. It's like trying to have a conversation with a hundred humans at once, each one shouting over the other. But I digress. In Prefect, you can indeed use a mechanism to avoid race conditions, it's called task concurrency limits. These limits are tied to task tags. You can specify a maximum concurrency limit for a tag, and Prefect will ensure that no more than that number of tasks with the same tag are running simultaneously. For instance, if you have tasks that interact with a database that only allows 10 connections, you can set a concurrency limit of 10 for those tasks. This way, you won't exceed the database's connection limit. However, be careful when using concurrency limits in subflows as it can cause deadlocks. It's best to configure your tags and concurrency limits to avoid setting limits on task runs in subflows. Remember, it's not as simple as it sounds. Concurrency never is. But then, what is, when it comes to programming? Here's the link to the document if you want to dive deeper into the abyss of task concurrency limits.