Josh08/18/2022, 9:56 PM
But I don’t want to overload my databases, so I want to set concurrency limits at the task level to make sure there are only ever
tasks running against a given database at one time. I know you can set task tags when I’m creating the task object, but can I do so as the flow is executing against mapped tasks?
N < limit
Matt Conger08/18/2022, 11:15 PM
Dylan08/18/2022, 11:21 PM
Dylan08/18/2022, 11:36 PM
Anna Geller08/19/2022, 12:00 AM
before mapping over it. This is already possible!
now assuming that your DB can only handle 10 parallel connections, you can limit concurrency using:
@task(tags=[db]) # this sets your tag "db"
this way, only 10 mapped child tasks will run concurrently/in parallel and your DB won't be overwhelmed All this is already possible, therefore I will close the issue
prefect concurrency-limit create db 10
Josh08/23/2022, 11:56 PM
queries = ["db1", "db2", "db1", "db2", "db3"] my_task.map(queries)
to get tags based on the parameters for which db it will run against. I don’t want to assign a static tag to
that will be the same across all parameters.
Anna Geller08/24/2022, 12:25 PM