Thanks for the detailed context — sorry you’re hitting this intermittently. Based on how the Cloud Run (Jobs) worker works in Prefect 3.x, this pattern is consistent with a known edge case: - Prefect’s Cloud Run V2 worker generates a unique Job name for each run as

<slugified-base>-<uuid4>

, so two concurrent runs should not collide on name. - However, during Job creation the Google API client can retry on transient network errors. If the first create request is accepted by GCP but the client retries (due to a transient error), the second attempt can receive a 409 AlreadyExists. Prefect’s worker does not currently treat 409 as “okay, the Job exists” — it treats it as an error. With

keep_job=False

, the worker will then attempt to delete the Job during error handling, which can make it look like the Job never existed and cause a failed submission. Code references (for transparency): - Unique job name generation (UUID suffix) in the Cloud Run V2 worker: job_name property - Error handling does not special-case 409 AlreadyExists (creation): _create_job_error - Error handling does not special-case 409 AlreadyExists (run submission): _job_run_submission_error - Deletion logic (when keep_job=False) that can immediately remove the Job on error: error path and cleanup after completion Why you see it “randomly” - Concurrency increases the chance of transient retries/409s surfacing. - With

timeout=0

(fire-and-forget) and

keep_job=False

, a create-then-409 scenario can delete the just-created Job, so you don’t see it in the UI by the time you look. What I’d like to confirm 1) Prefect and prefect-gcp versions, and the work pool type - Are you on Prefect 3.x and the “Cloud Run (V2)” work pool? - Please share:

prefect version
python -c "import prefect_gcp, pkgutil; import importlib.metadata as im; print('prefect-gcp', im.version('prefect-gcp'))"

2) Exact error text (including HTTP status code and resource path if present) from the worker logs for one failing submission. Mitigations you can try now - Temporarily enable

keep_job=True

to verify the behavior and aid debugging:

flow_run = await run_deployment(
    name=f"process-batch-subflow/{PREFECT_DEPLOYMENT_SUFFIX}-subflow",
    parameters={...},
    job_variables={
        "cpu": CPU_CORES,
        "memory": "32G",
        "keep_job": True,  # temporarily for debugging
    },
    timeout=0,
    tags=[model],
)

- Slightly stagger submissions or limit concurrency (reduces chances of transient retry collisions):

sem = anyio.Semaphore(2)  # or 1 to test serial
for model in collection_models:
    ...
    async with sem:
        await anyio.sleep(0.2)  # small jitter
        future = launch_model_subflow_task.with_options(
            name=f"launch-{model}",
            retries=2,
            retry_delay_seconds=60,
        ).submit(...)

- Upgrade to latest Prefect and prefect-gcp. If you’re not on the latest, there have been many Cloud Run worker improvements recently. Why this matters - There’s a gap in the Cloud Run V2 worker: it does not treat 409 AlreadyExists on create/run as a recoverable event. In an at-least-once request environment, the client can receive 409 even though the first create succeeded. Ideally the worker should detect 409 and proceed (or regenerate a new name and retry). Right now, it fails and, with

keep_job=False

, may delete the job. Next steps

- If you can share your versions and a failing log snippet, I can confirm and file an issue with our team (or I can open one on your behalf). If you prefer, I’ll include your anonymized details and these code references so we can add proper 409 handling in the worker. In the meantime, staggering submissions or temporarily setting

keep_job=True

should help verify this is the root cause and mitigate the impact.

@Marvin

(observatory) ➜  observatory git:(main) prefect version
Version:              3.4.19
API version:          0.8.4
Python version:       3.12.11
Git commit:           e7ae41cd
Built:                Fri, Sep 19, 2025 09:23 PM
OS/Arch:              darwin/arm64
Profile:              ephemeral
Server type:          cloud
Pydantic version:     2.11.7
Server:
  Database:           sqlite
  SQLite version:     3.43.2

I'm not using prefect_gcp, cloud run is just a config on the work pool but not in python.

Thanks — that helps. Even though you don’t import prefect-gcp in your code, the “Cloud Run (Jobs)” work pool uses the Cloud Run worker from prefect-gcp under the hood in the worker process. So the behavior I outlined still applies at submission time. What’s likely happening - When the worker creates a Cloud Run Job, the Google API client may retry on a transient error. If the first create succeeded but the client retries, the second attempt can hit 409 AlreadyExists. The current Cloud Run V2 worker does not special‑case 409; it treats it as a failure and, with keep_job=False, may immediately delete the just-created job. This makes it look like the job never existed although it briefly did. - Links to implementation for reference: - Unique job name generation with UUID: https://github.com/PrefectHQ/prefect-gcp/blob/main/prefect_gcp/workers/cloud_run_v2.py#L143-L171 - No special handling for 409 on create: https://github.com/PrefectHQ/prefect-gcp/blob/main/prefect_gcp/workers/cloud_run_v2.py#L794-L811 - No special handling for 409 on run submission: https://github.com/PrefectHQ/prefect-gcp/blob/main/prefect_gcp/workers/cloud_run_v2.py#L972-L999 - Cleanup when keep_job=False: https://github.com/PrefectHQ/prefect-gcp/blob/main/prefect_gcp/workers/cloud_run_v2.py#L566-L578 and https://github.com/PrefectHQ/prefect-gcp/blob/main/prefect_gcp/workers/cloud_run_v2.py#L755-L773 What I’d like from you to confirm - The exact work pool type (Cloud Run (Jobs) V2) and how/where your worker process runs. - The worker’s installed versions of prefect and prefect-gcp (from the worker environment). For example: - pip/pipx:

pip show prefect prefect-gcp

- A snippet from the worker logs showing the 409 error including the resource path (it usually includes the Job name). This will tell us if duplicate names are actually being generated, or if it’s a retried create that’s racing with itself. Mitigations you can try right now - Temporarily set keep_job=True in the work pool’s default variables or the deployment’s job variables. This will: - Let you see the Job in Cloud Run and its exact name when a 409 occurs. - Avoid immediate deletion that makes investigation difficult. - Add a small jitter/limit concurrency when submitting multiple deployments at once (reduces chance of client retry race): - e.g., a small

anyio.sleep(0.2-0.5)

between submits or a semaphore to 1–2 concurrent submits. - Ensure your worker is on the latest prefect and prefect-gcp. There have been Cloud Run worker fixes and improvements recently. If we confirm the 409 is a retried create (most likely), we should open an issue to add explicit 409 handling in the Cloud Run V2 worker so it treats “AlreadyExists” as a recoverable state during submission instead of failing and deleting. I’m happy to file that once we have the logs with the resource name. Separately, enabling keep_job=True just for a day while you reproduce will also prove whether names are indeed unique (they should be) and whether the failure correlates with immediate cleanup.