José Agretti
10/01/2025, 1:19 AMMarvin
10/01/2025, 1:19 AMJosé Agretti
10/01/2025, 1:27 AMMarvin
10/01/2025, 1:27 AMMarvin
10/01/2025, 1:27 AMflow.from_source(...).deploy(...), work pool name/type)
- Worker environment (sounds like Vertex AI in your setup) and any worker/flow logs
- Error messages or a flow run ID
- Any recent changes (e.g., Docker image/arch updates—you mentioned resolving an architecture mismatch before)
If this is specifically about your Vertex AI work pool or the Docker architecture issue you were troubleshooting, let me know where it’s stuck now and I’ll dive right in.José Agretti
10/01/2025, 1:28 AMMarvin
10/01/2025, 1:28 AMMarvin
10/01/2025, 1:30 AMprefect work-pool inspect <pool_name>
prefect work-pool set-concurrency-limit <pool_name> <int>
- Global concurrency limits (by key/tag)
- Enforced when your code acquires a slot (e.g., with concurrency("my-key")) or when tasks carry a tag that maps to a server concurrency limit.
- If the copied run is missing the expected key/tag (e.g., tags added dynamically from parameters or only set on the original), it won’t acquire the limit.
- Verify with:
prefect concurrency-limit ls
prefect concurrency-limit inspect <key_or_tag>
Check that the copied run’s tasks appear in active_slots.
- Ordering vs. enforcement
- Limits control how many run concurrently, not which one goes first. A copied run may start before another queued run once a slot frees up, but it should never exceed the configured capacity.
If you share:
- Which kind of limit you’re using (work pool limit vs. global concurrency key/tag)
- The original and copied flow run IDs
- The work pool name (I know you’re using Vertex AI pools)
…I can confirm whether the copied run acquired the correct slot and whether the pool limit was applied.
Docs for reference:
- Global concurrency limits
- Work pools and pool-level concurrencyMarvin
10/01/2025, 1:32 AMprefect work-pool inspect "<your-pool>"
Docs: CLI: work-pool
- Work queue
prefect work-queue inspect -p "<your-pool>" "<your-queue>"
Docs: CLI: work-queue
- Global concurrency limits
prefect concurrency-limit ls
Guide: Global concurrency limits
If you can share:
- Which exact limit you’re using (pool/queue/deployment/global)
- Prefect version
- An example where you see more concurrent runs than the limit (counts/time window)
- A flow run ID where a retry seemed to “skip” the limit
…I can dig in and help pinpoint what’s happening.José Agretti
10/01/2025, 1:41 AMMarvin
10/01/2025, 1:41 AMMarvin
10/01/2025, 1:44 AMprefect deployment inspect "<flow-name>/<deployment-name>"
- List current running runs for that deployment to compare against your limit:
prefect flow-run ls --flow-name "<flow-name>" --state RUNNING --limit 50
If you have an example where a retry started while the deployment was already at its max, share:
- The deployment name
- The “Max concurrent runs” value
- The flow run ID(s) for both the active runs and the retry that started
- Your Prefect version and pool (sounds like Vertex AI)
I’ll trace the states and confirm whether the retry re-acquired a slot as expected.
Related docs:
- Global concurrency limits (separate from deployment-level, but useful context)
- Work pools (for pool-level concurrency considerations)