Hello. I submitted 20 tasks to be executed with a concurrency limit of 4 (using tags). This results in no tasks being executed. All tasks are pending. Workers just spam this:

prefect-prefect-agent-3   | 11:17:06.370 | DEBUG   | prefect.engine - Received wait instruction for 30s: Concurrency limit for the max_4_concurrent tag has been reached

What am I doing wrong? For context: each task has to fetch a thing from the databse, send a POST request to an ML model, write a thing to the databse. I need no more than 4 tasks being executed in parllel, so that the model and the database are not overwhelmed.

Workers are do not execute tasks, they execute flow runs. https://docs.prefect.io/2.10.13/concepts/work-pools/#worker-overview 1. If you check your Prefect UI, are the tasks stuck at concurrency limit? Might have to delete the tag and try again.2. 2. What task runner are you using? The default uses Python threads which cause locking.

@Dominic Tarro I use the default runner. Should I switch to dask or ray?

I didn't read carefully enough. I don't think this is it. If your code triggers Python's GIL often then switching to those isn't a bad idea. However, if most of the task is just awaiting the database and model API, then that shouldn't be a huge problem. What are the task's parameter inputs? Does the task

wait_for

anything?

The task does not wait for anything. But I do wait for future results

labeling_task_ids = [task.id for task in labeling_tasks]

    <http://logger.info|logger.info>("Found %s labeling tasks for %s", len(labeling_tasks), date)

    futures = label_task.map(
        labeling_task_ids,
        db_uri,
        inference_api_url,
        exists_behavior,
    )

    results = [future.result() for future in futures]

What is

labeling_tasks

? Are those task runs?

@Dominic Tarro label_task are prefect task runs. labeling_task_ids are just parameters passed to these tasks (sorry for the naming confusion, these “tasks” have nothing to do with prefect tasks, they are just my internal entities from which I get some numeric ids)

Ahhh, okay. Yeah I'm not sure what's going on here sorry

Thanks for the help anyway