Josh
08/18/2022, 9:56 PMmy_task.map(databases)
But I don’t want to overload my databases, so I want to set concurrency limits at the task level to make sure there are only ever N < limit
tasks running against a given database at one time. I know you can set task tags when I’m creating the task object, but can I do so as the flow is executing against mapped tasks?Matt Conger
08/18/2022, 11:15 PMDylan
08/18/2022, 11:21 PMMarvin
08/18/2022, 11:22 PMDylan
08/18/2022, 11:36 PMAnna Geller
08/19/2022, 12:00 AMmy_task
before mapping over it. This is already possible!
@task(tags=[db]) # this sets your tag "db"
now assuming that your DB can only handle 10 parallel connections, you can limit concurrency using:
prefect concurrency-limit create db 10
this way, only 10 mapped child tasks will run concurrently/in parallel and your DB won't be overwhelmed
All this is already possible, therefore I will close the issueJosh
08/23/2022, 11:56 PMqueries = ["db1", "db2", "db1", "db2", "db3"]
my_task.map(queries)
my_task
to get tags based on the parameters for which db it will run against. I don’t want to assign a static tag to my_task
that will be the same across all parameters.Anna Geller
08/24/2022, 12:25 PM