https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> when i do prefect deploy in gcp cloud...
# marvin-ai
a
@Marvin when i do prefect deploy in gcp cloud job v2 run, only 8 flows get deployed, and the rest error with: the thing is no matter what i do, deploy sequentially or deploy it parallel, it always only deploy a maximum of fixed 8 flows, the rest of the flow deployment fails DEFAULT 2026-04-23T090243.434101Z 090243.431 | ERROR | uvicorn.error - [Errno 98] error while attempting to bind on address ('127.0.0.1', 8781): [errno 98] address already in use DEFAULT 2026-04-23T090243.440664Z 090243.438 | ERROR | uvicorn.error - Traceback (most recent call last): DEFAULT 2026-04-23T090243.440752Z assert self.consumer_task is not None, "Reactive triggers not started" DEFAULT 2026-04-23T090243.440756Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ DEFAULT 2026-04-23T090243.440771Z AssertionError: Reactive triggers not started DEFAULT 2026-04-23T090243.440900Z 090243.439 | ERROR | uvicorn.error - Application shutdown failed. Exiting. ERROR 2026-04-23T090252.037455Z [severity: ERROR] Traceback (most recent call last): File "/root/prefect-deploy-flows.py", line 904, in <module> main() ~~~~^^ File "/root/prefect-deploy-flows.py", line 900, in main deploy_deployments(env) ~~~~~~~~~~~~~~~~~~^^^^^ File "/root/prefect-deploy-flows.py", line 863, in deploy_deployments asyncio.run(async_deploy_deployments(env)) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.13/asyncio/runners.py", line 195, in run return runner.run(main) ~~~~~~~~~~^^^^^^ File "/usr/local/lib/python3.13/asyncio/runners.py", line 118, in run return self._loop.run_until_complete(task) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/usr/local/lib/python3.13/asyncio/base_events.py", line 725, in run_until_complete return future.result() ~~~~~~~~~~~~~^^ File "/root/prefect-deploy-flows.py", line 856, in async_deploy_deployments await asyncio.gather(*(deploy_single_flow(info) for info in flows)) File "/root/prefect-deploy-flows.py", line 851, in deploy_single_flow deployment = await flow_obj.to_deployment(**to_deploy_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.13/site-packages/prefect/flows.py", line 803, in ato_deployment return await RunnerDeployment.afrom_storage( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ DEFAULT 2026-04-23T090252.037465Z ...<22 lines>... DEFAULT 2026-04-23T090252.037469Z ) DEFAULT 2026-04-23T090252.037472Z ^ DEFAULT 2026-04-23T090252.037476Z File "/opt/venv/lib/python3.13/site-packages/prefect/deployments/runner.py", line 1224, in afrom_storage DEFAULT 2026-04-23T090252.037479Z await storage.pull_code() DEFAULT 2026-04-23T090252.037482Z File "/opt/venv/lib/python3.13/site-packages/prefect/runner/storage.py", line 417, in pull_code DEFAULT 2026-04-23T090252.037488Z await self._clone_repo() DEFAULT 2026-04-23T090252.037491Z File "/opt/venv/lib/python3.13/site-packages/prefect/runner/storage.py", line 470, in _clone_repo DEFAULT 2026-04-23T090252.037504Z raise RuntimeError(error_message) from exc_chain DEFAULT 2026-04-23T090252.040382Z RuntimeError: Failed to clone repository 'https://github.com/Armor1AI/armor1-apps.git' with exit code 1.
m
✅ thought for 234.9 seconds
This looks like ephemeral Prefect server startup is colliding during deployment creation, not a hard “8 flow” deployment limit. The key clue is this:
Copy code
[Errno 98] error while attempting to bind on address ('127.0.0.1', 8781): address already in use
AssertionError: Reactive triggers not started
and then later:
Copy code
File ".../prefect/deployments/runner.py", line 1224, in afrom_storage
  await storage.pull_code()
...
RuntimeError: Failed to clone repository '<https://github.com/Armor1AI/armor1-apps.git>'
What’s happening When Prefect creates deployments programmatically with 
flow.to_deployment()
/ 
flow.ato_deployment()
and there is no explicit 
PREFECT_API_URL
, Prefect can spin up a temporary local API server in-process / subprocess. That temporary server uses a localhost port in the 8000–9000 range. Your logs show one of those attempts trying 
127.0.0.1:8781
and failing because it’s already in use. Once that startup fails, shutdown can also error with:
Copy code
AssertionError: Reactive triggers not started
So the “only 8 flows” symptom is likely a side effect of repeated deployment creation causing server/process/port contention in your Cloud Run Job container, not an actual supported limit of 8. What I’d recommend first 1. Set 
PREFECT_API_URL
explicitly to your Prefect Cloud workspace API URL before running deployment creation. - This avoids ephemeral local server creation entirely. - Prefect Cloud docs: Connect to Prefect Cloud 2. Upgrade to the latest Prefect 3.x - There has been work/fixes around ephemeral server lifecycle/cleanup. - Release notes: Prefect releases 3. Avoid creating many deployments via concurrent 
ato_deployment()
calls - Even if you tried sequential mode, if each call creates storage and client state independently inside the same job process, you can still run into cleanup timing issues. - Better to batch deployment registration through a single API-backed process. 4. Avoid repeated Git clone per flow if all flows come from the same repo - Your traceback shows failure inside 
afrom_storage()
→ 
pull_code()
→ 
_clone_repo()
. - If every flow points at the same Git repo URL, repeated deployment creation may be repeatedly cloning/fetching the same repo under contention. Most likely root cause chain - 
flow.ato_deployment(...)
- Prefect needs a client / deployment metadata handling - no 
PREFECT_API_URL
available, so an ephemeral API server is started - one of those startup attempts binds localhost port 
8781
- bind fails with 
Errno 98
- server startup/shutdown leaves the “Reactive triggers not started” assertion - the deployment call then also fails while resolving storage / cloning repo Concrete fixes If you are using Prefect Cloud, make sure the job has:
Copy code
PREFECT_API_URL=<https://api.prefect.cloud/api/accounts/<account-id>/workspaces/<workspace-id>>
PREFECT_API_KEY=<your-api-key>
If you are self-hosting Prefect server, set:
Copy code
PREFECT_API_URL=http://<your-prefect-server-host>:4200/api
Also, if you want to fail fast instead of silently starting an ephemeral API:
Copy code
PREFECT_SERVER_ALLOW_EPHEMERAL_MODE=false
That helps confirm whether ephemeral mode is the culprit. If you’re deploying from Python code Try to avoid per-flow 
ato_deployment()
calls that each resolve source storage from Git separately. If all flows live in one checked-out repo already inside the container, prefer loading them from the local filesystem instead of Git-based storage for deployment creation. Also, if you can share code, I’d specifically look at whether you are doing something like:
Copy code
await asyncio.gather(*(deploy_single_flow(info) for info in flows))
because that is especially likely to trigger this. Why the fixed number looks like 8
I don’t think Prefect has a documented “max 8 deployments” limit. In Cloud Run Job, the repeatable “8 succeed, rest fail” usually points to some resource/process/socket contention pattern in the container rather than a Prefect deployment cap. Relevant docs - Deployments - Prefect server - Connect to Prefect Cloud - Prefect releases Important note on your traceback The final exception shown is the Git clone failure:
Copy code
RuntimeError: Failed to clone repository '<https://github.com/Armor1AI/armor1-apps.git>'
That may be secondary, but it’s still worth checking: - GitHub auth / rate limiting - concurrent clone attempts to same destination - temp disk limits in Cloud Run Job - whether the repo is already present locally and can be used directly instead of cloning for every flow If you want, paste: - your Prefect version - whether 
PREFECT_API_URL
is set - the code around 
deploy_single_flow
- whether these flows all use the same GitHub source and I can suggest a safer deployment pattern.
a
@Marvin i am setting the prefect url, how else do i see the deployment actually get reflected in the remote prefect server, its actually weird that its even trying to spin up an ephemeral server. if that was truely the case, then i shouldnt see the deployments or its changes reflect in the remove prefect server. @Nate sorry for the ping, but isnt this weird?
m
✅ thought for 190.9 seconds
Yes — it is weird, and your instinct is right. If deployments are actually being created/updated in the remote Prefect server, then the deployment code is reaching your remote API successfully for at least part of the workflow. So seeing 
uvicorn.error
and a localhost bind failure at the same time suggests one of these is happening: 1. some code path is still trying to start a temporary local Prefect server, or 2. a subprocess / context is not seeing the same 
PREFECT_API_URL
you think it is, even though the main deployment path is. So: yes, weird — but still possible. Why both can be true at once A deployment operation is not always one single API call. During deployment creation, Prefect may: - load flow metadata - resolve storage/source - clone/pull code - validate pool/infrastructure config - create/update deployment in the API So you can absolutely have a situation where: - some steps hit the remote Prefect API successfully - another step, possibly in a different settings context or subprocess, still falls back to ephemeral/local server behavior That would explain why you do see changes reflected remotely and still get localhost 
uvicorn
errors. The most likely explanation In Prefect, 
get_client()
checks the current resolved settings, not just “is there an env var somewhere in the container.” If the active settings context for a particular call/subprocess resolves 
PREFECT_API_URL
as empty/falsy, Prefect may attempt ephemeral mode. That means all of these are possible: - your top-level process has 
PREFECT_API_URL
- one subprocess does not - one temporary settings context overrides it - one code path is launched with different env inheritance - some child process logs 
uvicorn.error
even while the main process is talking to Prefect Cloud That would match your symptoms exactly. Important detail from your traceback Your failure is happening here:
Copy code
flow_obj.to_deployment(...)
...
RunnerDeployment.afrom_storage(...)
...
await storage.pull_code()
...
Failed to clone repository ...
That suggests this is not purely “deployment registration” logic; it’s also interacting with storage/source loading. If your flows are built from a Git source, that path may be creating additional subprocess/context behavior. So there may actually be two overlapping problems: 1. unexpected local ephemeral server startup attempt 2. Git clone/storage contention/failure during 
afrom_storage()
 What I would check next Please add this right before the failing deployment call and log it from the same function/process that calls `to_deployment`:
Copy code
import os
from prefect.settings import PREFECT_API_URL, PREFECT_SERVER_ALLOW_EPHEMERAL_MODE

print("ENV PREFECT_API_URL =", os.getenv("PREFECT_API_URL"))
print("RESOLVED PREFECT_API_URL =", PREFECT_API_URL.value())
print("EPHEMERAL MODE =", PREFECT_SERVER_ALLOW_EPHEMERAL_MODE.value())
If possible, also log this inside any subprocess entrypoint if you have one. What I’m looking for: - if 
os.getenv("PREFECT_API_URL")
is set but 
PREFECT_API_URL.value()
is empty, then there is a settings-context override - if both are set, but Prefect still tries ephemeral, then we need to identify the exact subprocess/code path doing it - if the child process has no env var, then it’s env inheritance Very useful defensive setting Try forcing this:
Copy code
PREFECT_SERVER_ALLOW_EPHEMERAL_MODE=false
If the weird localhost server attempt is real, that should either: - stop it entirely, or - turn the behavior into a clearer hard failure telling us which path is missing API config That’s a very good debugging move here. Why I don’t think your observation disproves ephemeral mode You said:
if that was truely the case, then i shouldnt see the deployments or its changes reflect in the remote prefect server
Not necessarily. A run can: - create/update deployments 1–8 remotely just fine
- then a later deployment or subprocess tries local ephemeral startup and fails - so you still see successful remote changes for the earlier subset Or even within one deployment attempt: - remote API call succeeds - later storage/source validation step spins up something local and fails - net result is partial success / inconsistent logs So your observation is consistent with a mixed execution path, not necessarily a contradiction. What I suspect most strongly in your case Given: - only a subset succeeds - failure appears during 
afrom_storage()
/ 
pull_code()
/ clone - 
uvicorn.error
shows up - this runs in GCP Cloud Run Job I suspect one of: - concurrent or repeated Git-based source resolution is spawning code paths with different settings - a subprocess used during deployment creation is not inheriting 
PREFECT_API_URL
- a Prefect bug/regression in deployment/storage handling is causing an unintended ephemeral server startup despite remote API configuration Practical next steps 1. Set and log:
Copy code
PREFECT_API_URL
PREFECT_API_KEY
PREFECT_SERVER_ALLOW_EPHEMERAL_MODE=false
2. Log both:
Copy code
os.getenv("PREFECT_API_URL")
PREFECT_API_URL.value()
3. Confirm whether all flows are coming from: - 
flow.from_source(...)
with the same Git repo, or - locally imported Python modules already present in the container 4. If the repo is already in the container, avoid recloning from Git during deployment creation if possible. 5. Upgrade to the latest Prefect 3.x if you haven’t already: Prefect releases Docs / references - Connect to Prefect Cloud - Deployments - Prefect server - Prefect releases If you want, paste: - the code that constructs 
flow_obj
- the exact 
to_deployment(...)
/ 
deploy(...)
call - whether you use 
from_source("<https://github.com/...>")
- output of the three debug prints above and I can help pinpoint whether this is settings leakage, subprocess env inheritance, or the Git storage path specifically.
n
huh, let me take a look
👍 1
i think i reproduced this, a shared state mutation bug if i have it right. will link a PR shortly
a
yeah i did think its some file collision issue, but i did observe this even when i try to run it sequentially. Also one thing to note is if i run the script locally to deploy, it works without raising the clone issue. I have only observed this in GCP cloud run jobs, and was never able to recreate the issue locally
also any work around until this issue is solved? i have tried everything, using phased deployment 4 at a time per client context and not reusing the client context (by creating new client for every flow), nothing seems to work
i even tried using a different GitRepository instance, but that fails too
n
huh. id guess there's some GCP resource exhaustion of some kind going on then
ah wait
uvicorn.error - [Errno 98] error while attempting to bind on address ('127.0.0.1', 8781): [errno 98] address already in use
are you still seeing this?
/ are you still seeing weird things that look like an ephemeral prefect server interfering? is your expectation that you have a long-lived prefect server you're doing these things against?
a
no no the uvicorn issue was a dumb thing i did when i was trying to fix this issue, thats no longer there
n
oh ok thanks
a
the clone failed is the only issue
huh. id guess there's some GCP resource exhaustion of some kind going on then
@Nate does that mean each flow does not cleanup after itself? the number 8 seems to be too specific right, it clones 8 times, the 9th time there is no space?
n
i don't see any evidence of that, do you? i'd like to see what's going on with that exit code 1, since i can't really reproduce this against a sandbox GCP project i just spun up https://github.com/zzstoatzz/prefect-git-clone-gcp-diagnostics
not sure if you've got the time or patience to try, but i think if you just use your repo url and entrypoint, this should spit out the git stderr that's getting swallowed now
a
i don't see any evidence of that, do you?
no i did dig into the code, and the flows do cleanup, its inside the TemporaryDirectory context which auto clears the cloned repo, however i do suspect it might be some kind of bytecode cache of mmap references via sys.modules. i think the imported modules are not cleared even tho the folders are
i'd like to see what's going on with that exit code 1
for some other unrelated reason, i did observe that the reason of clone failure is masked by prefect, i might be wrong, but my understanding was that twas to prevent leaking the secrets if the actual git clone failure log is shown, which will include the credentials
maybe if i can use a patched prefect distribution which shows the exact clone failure reason, i will be able to get to the bottom of this, will try it out and get back
n
hmm you might be right about that module leak
prefect rocket 1
a
@Nate so i shipped with a patched prefect distribution which exposes the real cause, the issue was fanning out calls, the repo actually has nested modules and each clone has about 3 sub modules. the GCP NAT was the bottleneck which blocked the concurrent requests. surprisingly this time when i did sequential with sleep it worked, so not a prefect related issue ig. have solved the issue for now
i did soo many changes to the deployment script that idk what worked
n
GCP NAT was the bottleneck which blocked the concurrent requests
gotcha, i was wondering if something like that was going on
huh. id guess there's some GCP resource exhaustion of some kind going on then
glad you have something working! thanks for helping expose the module leak at least!
a
yee, thanks for your help too looking into this! this was bothering me for a while now as it kept breaking my CI/CD
4 Views
Open in Slack
PreviousNext
Powered by