https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I have configured a Task Run concurre...
# ask-marvin
j
@Marvin I have configured a Task Run concurrency limit:
Copy code
Tag
db-sheets-github-sync
Concurrency Limit Count
5
Concurrency Limit Active Task Runs Count
5
Concurrency Limit ID
984af7c0-e1e3-40e1-85e9-29ee8bc10722
Created
2025/10/16 01:26:24 PM
Updated
2025/10/16 03:56:54 PM
My code refers to the tag as follows:
Copy code
@task(tags=["db-sheets-github-sync"])
def sync_job_to_github_task(job_uuid: str, clone_path: str = None):
When the flow runs, it spawns 10 tasks simultaneously. They all time out because they overwhelm the database. Have I misconfigured something? How should I expect to see the concurrency limit reflected in the logs?
m
✅ thought for 343.9 seconds
Short answer: with a task tag concurrency limit of 5, Prefect Cloud should only allow 5 task runs with that tag to enter RUNNING at a time. The next tasks should wait in a scheduled “AwaitingConcurrencySlot” state and not execute code until a slot is free. You should not see 10 actually running at once. What you should see when the limit is working - Task run states: extra tasks remain SCHEDULED with state name “AwaitingConcurrencySlot”. - Worker logs: periodic messages that the task is waiting for a concurrency slot and will retry in ~30 seconds. - Only up to 5 task logs should show your task’s “start” log line at any given time. How to verify quickly 1) Inspect the limit and active slots
Copy code
prefect concurrency-limit inspect db-sheets-github-sync
- Expect 
limit: 5
and 
active_slots
listing up to 5 task run IDs. 2) List task runs that are waiting on slots
Copy code
prefect task-run ls --state "AwaitingConcurrencySlot" --limit 50
- You should see the “extra” task runs here when 5 are already RUNNING. 3) Check that your task runs really have the tag - In the Cloud UI, open a task run and look at its Tags. - Or via CLI, pick a run ID and:
Copy code
prefect task-run inspect <task-run-id>
- Confirm 
db-sheets-github-sync
is present. Common causes when tasks exceed the limit - Tag mismatch: a typo, case difference, or the tag not actually present on the task run (e.g., overridden via 
.with_options(tags=...)
). - Tasks called directly instead of submitted: only orchestrated task runs are limited. Use 
my_task.submit(...)
for concurrent execution; calling 
my_task(...)
inside a flow runs synchronously in-process (still orchestrated, but won’t contribute to “spawned” concurrency the same way). - Limit created after tasks were already RUNNING: concurrency is enforced at transition to RUNNING; existing RUNNING tasks aren’t preempted. - They aren’t tasks: subflows are not governed by task tag concurrency limits. - Rare slot issues: if 
inspect
shows more than 5 
active_slots
or shows none while many are RUNNING, capture details. Troubleshooting steps I recommend now - Run:
Copy code
prefect concurrency-limit inspect db-sheets-github-sync
prefect task-run ls --state-type RUNNING --limit 50
prefect task-run ls --state "AwaitingConcurrencySlot" --limit 50
- Confirm the tag is on those task runs (UI or 
prefect task-run inspect <id>
). - If slots look “stuck”, you can clear them:
Copy code
prefect concurrency-limit reset db-sheets-github-sync
- If you want more visibility in worker logs, set 
PREFECT_LOGGING_LEVEL=DEBUG
on the worker. If you can share a flow run URL where this happened, I can look for the “AwaitingConcurrencySlot” states and verify the tags and slot accounting. Docs for reference - Tag-based concurrency limits: Docs: Tag-based concurrency limits - CLI for concurrency limits: CLI: concurrency-limit - States concept (incl. “AwaitingConcurrencySlot”): Concepts: States
j
@Marvin How would this behavior be impacted by mapping the arguments, e.g.
Copy code
github_sync_futures = sync_job_to_github_task.map(
                job_uuid=uuids_needing_sync,
                clone_path=[clone_path_str] * len(uuids_needing_sync)
            )
m
✅ thought for 48.2 seconds
Mapping doesn’t bypass tag-based concurrency limits. Each mapped item becomes its own task run with the same tags, so the limit is enforced across all of them. What to expect with your example - If you map 10 items and the tag limit is 5, up to 5 child task runs should move to RUNNING; the rest should stay SCHEDULED with state name “AwaitingConcurrencySlot” until a slot frees up. - Your task code should not start executing for the “AwaitingConcurrencySlot” runs. You’ll only see your task’s “start” logs for at most 5 at a time. How to verify with mapped runs - Inspect the limit and active slots:
Copy code
prefect concurrency-limit inspect db-sheets-github-sync
- List waiting runs:
Copy code
prefect task-run ls --state "AwaitingConcurrencySlot" --limit 50
- Spot-check a mapped child’s tags:
Copy code
prefect task-run inspect <task-run-id>
Confirm 
db-sheets-github-sync
is present. Gotchas to watch for - Tag mismatch: ensure the mapped runs really carry 
db-sheets-github-sync
(e.g., not overridden via 
with_options
). - Existing RUNNING tasks: if 5 are already running from earlier, all new mapped runs will wait. - Worker logs: you’ll see periodic messages about waiting for a slot; no task code runs until the slot is granted. If you still observe all 10 mapped tasks executing simultaneously, that points to a tag/config mismatch or an orchestration issue. Share a flow run URL and I’ll check for “AwaitingConcurrencySlot” states and the tag presence across mapped children. Docs for reference - Tag-based limits: How-to: Tag-based concurrency limits - CLI: CLI: concurrency-limit - States (incl. “AwaitingConcurrencySlot”): Concepts: States
Open in Slack
PreviousNext
Powered by