Sorry for the delay! I don’t see the original question in this thread. Could you paste it here or give a quick summary of what you’re trying to do? Helpful details to include so I can jump on it right away: - Prefect version (2.x or 3.x) - How you’re running (local, deployment) and infrastructure (Process, Docker, Kubernetes) - Worker type and work pool name (if using deployments) - Storage/source (e.g., from_source GitHub/S3) and any relevant blocks/secrets - Error message or logs (top stack trace lines) - Minimal code snippet or steps to reproduce If you’re on Prefect 2.x, note that I’ll default guidance to 3.x unless you prefer sticking with 2.x.

Got it — when you see “first 4 run, then the rest stay Pending,” it’s almost always a concurrency/capacity gate somewhere. A few common culprits in Prefect 3.x and quick checks: Likely causes to check - Tag-based concurrency limits - If your spider task has a tag with limit=4, the next batch will sit in Pending (or AwaitingConcurrencySlot) until slots free up. If the limit is a rate limit, it may wait for tokens to replenish. - Check the Concurrency Limits page and the state details on a Pending task to see if it says “waiting for concurrency slot.” - Docs: Tag-based concurrency and Global concurrency limits - Work pool/worker capacity when launching child flow runs - If your “spiders” are launched as separate flow runs (e.g., via

run_deployment

or API), they will wait for a worker slot. If your worker concurrency is 4 and it’s also running the parent flow, the remaining child runs can sit Pending until there’s capacity. - Fixes: - Increase worker concurrency/capacity for the pool - Run child work in a separate pool from the parent - Or run spiders as tasks/subflows within the same process instead of separate flow runs - Docs: Work pools - Task runner concurrency vs orchestration limits - If you’re relying on

ConcurrentTaskRunner(max_workers=4)

, it should continue scheduling more work as threads free up. If tasks still hang, it’s usually an orchestrator-level limit (tags/global) or you’re awaiting too early (see next bullet). - Docs: Task runners - Awaiting too early inside the loop - If your code awaits each spider immediately in the loop, you’ll effectively serialize after the first batch. Prefer submitting all, then waiting: - Example pattern:

from prefect import flow, task
    from prefect.task_runners import ConcurrentTaskRunner

    @task(tags=["spider"])
    def run_spider(name):
        ...

    @flow(task_runner=ConcurrentTaskRunner(max_workers=4))
    def run_all(spiders: list[str]):
        futures = [run_spider.submit(s) for s in spiders]
        return [f.result() for f in futures]

- If you’re using tag-based concurrency for “spider”, set that limit in UI to 4 and Prefect will keep backfilling as slots free. - Rate limits vs fixed concurrency - If you used a rate limit (global or tag-based) rather than a simple concurrency limit, new tasks may remain scheduled/pending until the refill rate allows more. Check the state message for “waiting for rate limit” language. What would help me pinpoint it - Are the spiders tasks in one flow, or separate flow runs kicked off from the parent (e.g., `run_deployment`/API)? - How did you set “concurrency 4”? Task runner max_workers, tag-based concurrency, or worker/pool capacity? - A small code snippet of how you launch the spiders (loop, submit/map, await). - From the UI, what does a Pending spider’s state details say? (“Awaiting concurrency slot”, “Scheduled by …”, etc.) If you can share those details, I can give an exact fix. In the meantime, check the Pending task’s state message and the Concurrency Limits page — they usually reveal whether it’s a tag/global limit or worker capacity.