https://prefect.io logo
Join Slack
Powered by
This message was deleted.
# marvin-ai
s
This message was deleted.
m
Your message was too long, here's your message at the allowed limit: when i do prefect deploy in gcp cloud job v2 run only 8 flows get deployed and the rest error with the thing is no matter what i do deploy sequentially or deploy it parallel it always only deploy a maximum of fixed 8 flows the rest of the flow deployment fails DEFAULT 2026-04-23T09 02 43 434101Z 09 02 43 431 | ERROR | uvicorn error - [Errno 98] error while attempting to bind on address ('127 0 0 1' 8781) [errno 98] address already in use DEFAULT 2026-04-23T09 02 43 440664Z 09 02 43 438 | ERROR | uvicorn error - Traceback (most recent call last) DEFAULT 2026-04-23T09 02 43 440674Z File /opt/venv/lib/python3 13/site-packages/starlette/routing py line 694 in lifespan DEFAULT 2026-04-23T09 02 43 440678Z async with self lifespan_context(app) as maybe_state DEFAULT 2026-04-23T09 02 43 440682Z ~~~~~~~~~~~~~~~~~~~~~^^^^^ DEFAULT 2026-04-23T09 02 43 440686Z File /usr/local/lib/python3 13/contextlib py line 221 in aexit DEFAULT 2026-04-23T09 02 43 440690Z await anext(self gen) DEFAULT 2026-04-23T09 02 43 440695Z File /opt/venv/lib/python3 13/site-packages/prefect/server/api/server py line 730 in lifespan DEFAULT 2026-04-23T09 02 43 440700Z async with AsyncExitStack() as stack DEFAULT 2026-04-23T09 02 43 440704Z ~~~~~~~~~~~~~~^^ DEFAULT 2026-04-23T09 02 43 440709Z File /usr/local/lib/python3 13/contextlib py line 768 in aexit DEFAULT 2026-04-23T09 02 43 440713Z raise exc DEFAULT 2026-04-23T09 02 43 440717Z File /usr/local/lib/python3 13/contextlib py line 751 in aexit DEFAULT 2026-04-23T09 02 43 440721Z cb_suppress = await cb(*exc_details) DEFAULT 2026-04-23T09 02 43 440726Z ^^^^^^^^^^^^^^^^^^^^^^ DEFAULT 2026-04-23T09 02 43 440730Z File /usr/local/lib/python3 13/contextlib py line 221 in aexit DEFAULT 2026-04-23T09 02 43 440735Z await anext(self gen) DEFAULT 2026-04-23T09 02 43 440739Z File /opt/venv/lib/python3 13/site-packages/prefect/server/services/base py line 100 in running DEFAULT 2026-04-23T09 02 43 440744Z await asyncio gather(*[service stop() for service in service_tasks]) DEFAULT 2026-04-23T09 02 43 440748Z File /opt/venv/lib/python3 13/site-packages/prefect/server/events/services/triggers py line 58 in stop DEFAULT 2026-04-23T09 02 43 440752Z assert self consumer_task is not None Reactive triggers not started DEFAULT 2026-04-23T09 02 43 440756Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ DEFAULT 2026-04-23T09 02 43 440771Z AssertionError Reactive triggers not started DEFAULT 2026-04-23T09 02 43 440900Z 09 02 43 439 | ERROR | uvicorn error - Application shutdown failed Exiting ERROR 2026-04-23T09 02 52 037455Z [severity ERROR] Traceback (most recent call last) File /root/prefect-deploy-flows py line 904 in <module> main() ~~~~^^ File /root/prefect-deploy-flows py line 900 in main deploy_deployments(env) ~~~~~~~~~~~~~~~~~~^^^^^ File /root/prefect-deploy-flows py line 863 in deploy_deployments asyncio run(async_deploy_deployments(env)) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /usr/local/lib/python3 13/asyncio/runners py line 195 in run return runner run(main) ~~~~~~~~~~^^^^^^ File /usr/local/lib/python3 13/asyncio/runners py line 118 in run return self _loop run_until_complete(task) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File /usr/local/lib/python3 13/asyncio/base_events py line 725 in run_until_complete return future result() ~~~~~~~~~~~~~^^ File /root/prefect-deploy-flows py line 856 in async_deploy_deployments await asyncio gather(*(deploy_single_flow(info) for info in flows)) File /root/prefect-deploy-flows py line 851 in deploy_single_flow deployment = await <http //flow_obj to|flow_obj to>_deployment(**to_deploy_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /opt/venv/lib/python3 13/site-packages/prefect/flows py line 803 in ato_deployment return await RunnerDeployment afrom_storage( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ DEFAULT 2026-04-23T09 02 52 037465Z <22 lines> DEFAULT 2026-04-23T09 02 52 037469Z ) DEFAULT 2026-04-
23T09 02 52 037472Z ^ DEFAULT 2026-04-23T09 02 52 037476Z File /opt/venv/lib/python3 13/site-packages/prefect/deployments/runner py line 1224 in afrom_storage DEFAULT 2026-04-23T09 02 52
✅ thought for 173.8 seconds
This looks like a real Prefect/runtime issue, not something wrong with your deployment loop. What stands out from your logs is: - 
address already in use
on 
127.0.0.1:8781
- 
AssertionError: Reactive triggers not started
- failures inside 
Flow.ato_deployment()
/ 
RunnerDeployment.afrom_storage()
- then 
Failed to clone repository ...
That combination strongly suggests that during deployment Prefect is spinning up local temporary server/process resources, and when you deploy many flows in the same job/container those resources are colliding or not cleaning up correctly. The repo clone failures are likely a downstream symptom once the local deployment machinery gets into a bad state. A few important notes: 1. The fixed number of 8 is a clue If it always tops out around 8, that usually means some bounded local resource is being exhausted or contended: - local subprocesses - temp server instances / ports - clone destinations / locks - event/reactive trigger startup/shutdown state 2. The 
8781
bind error is likely the smoking gun Something in the deployment path is trying to start a local server on a loopback port, and subsequent deployments are colliding on it. 3. Parallel vs sequential not helping That implies the issue is not just your 
asyncio.gather(...)
, but that repeated deployments in the same process/container are reusing shared local state. I also checked the current CLI shape, and 
prefect deploy
is the correct 3.x command: - Prefect docs - CLI confirms 
prefect deploy [ENTRYPOINT]
What I’d recommend trying first: Best workaround: avoid per-flow source cloning / local storage resolution repeatedly in the same process Instead of calling 
flow_obj.to_deployment()
/ 
ato_deployment()
many times against the same Git source in one job, prefer one of these approaches: Option A: use 
prefect deploy
from a project with 
prefect.yaml
 If your flows are already in the container filesystem, deploy from local code instead of making each deployment re-pull the repo. Option B: use 
flow.from_source(...).deploy(...)
once per flow, but from a script that is not doing massive concurrent deployment And if possible, split the deployment batch into smaller separate job invocations. Option C: if you're on Prefect Cloud, make sure 
PREFECT_API_URL
points directly to Cloud That avoids any fallback behavior that may be starting local ephemeral server components. Things to verify in your Cloud Run Job environment:
Copy code
bash
echo $PREFECT_API_URL
echo $PREFECT_API_KEY
prefect config view
You want 
PREFECT_API_URL
to be your Prefect Cloud workspace API, not a local URL. A practical mitigation plan: 1. Upgrade Prefect to the latest 3.x available There have been recent fixes around deployment/storage concurrency behavior. 2. Stop deploying all flows in one Python process Even if you do it sequentially, the shared process may retain bad state. Try: - one 
prefect deploy
per process - or batches of, say, 5 flows per job 3. Avoid 
ato_deployment()
against the same git repo repeatedly if the code is already present locally Your traceback shows: - 
afrom_storage
- 
storage.pull_code()
- 
_clone_repo()
That means each deployment is going through storage cloning logic. If the code is already in the job image, that's unnecessary and more fragile. 4. Try deploying from local source instead of git source inside the deployment step For example, use a normal project deployment layout and run:
Copy code
bash
prefect deploy --all
from the checked-out project directory. 5. If you must keep the Python deploy script, avoid 
asyncio.gather
and isolate each deployment in a subprocess Not ideal, but it prevents shared in-memory state collisions. Example pattern: ```python import subprocess import sys flows = [ "./flows/a.py:flow_a", "./flows/b.py:flow_b", ] for entrypoint in flows: result = subprocess.run( ["prefect", "deploy", entrypoint],
check=False, text=True, ) if result.returncode != 0: print(f"Failed deploying {entrypoint}") ``` That is slower, but much safer than many 
ato_deployment()
calls in one interpreter. If you want, I can help you refactor your current script away from this pattern. Two likely root causes in your specific setup: - shared local server state / port collision causing the 
8781
bind failure - repeated git clone/storage resolution during 
afrom_storage()
causing the later repo clone errors If you can share: - your Prefect version - whether you're using Prefect Cloud or Server - how each flow is being constructed before 
to_deployment()
- whether the repo is already present in the Cloud Run image I can suggest the safest exact deployment pattern for your setup. Relevant links: - Deploy via Python - Deploy with prefect.yaml - Flow API docs - Deployments overview If you want, paste the part of your script where you build 
flow_obj
and call 
to_deployment()
, and I’ll show you the least invasive fix.
Open in Slack
PreviousNext
Powered by