https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> in cloud run work pool how do i speci...
# ask-marvin
a
@Marvin in cloud run work pool how do i specify
Copy code
"name": secret_base_name,
"valueFrom": {
    "secretKeyRef": {
        "name": final_secret_name,
        "key": "latest"
    }
}
secret key ref based env vars in the job_body
m
✅ thought for 459.4 seconds
Great question. The correct shape depends on which Cloud Run worker your work pool uses: - Cloud Run v2 worker: use valueSource.secretKeyRef with secret/version - Legacy Cloud Run (v1/Knative) worker: use valueFrom.secretKeyRef with name/key Most Prefect 3 Cloud Run pools use the v2 worker. Here’s how to specify a Secret Manager reference in your job config. Option A — set via job_variables (per deployment)
Copy code
json
{
  "env": [
    {
      "name": "<ENV_NAME>",
      "valueSource": {
        "secretKeyRef": {
          "secret": "<SECRET_MANAGER_NAME>",
          "version": "latest"
        }
      }
    }
  ]
}
Option B — bake it into the pool’s base job template (job_body) For a Cloud Run Job v2 template, env goes under template.template.containers[0].env:
Copy code
json
{
  "template": {
    "template": {
      "containers": [
        {
          "env": [
            {
              "name": "<ENV_NAME>",
              "valueSource": {
                "secretKeyRef": {
                  "secret": "<SECRET_MANAGER_NAME>",
                  "version": "latest"
                }
              }
            }
          ]
        }
      ]
    }
  }
}
You can update the pool template with:
Copy code
bash
prefect work-pool update "<your-pool>" --base-job-template ./base_job_template.json
If your pool is the legacy Cloud Run (v1) worker, use the Kubernetes-style schema you posted:
Copy code
json
{
  "name": "<ENV_NAME>",
  "valueFrom": {
    "secretKeyRef": {
      "name": "<SECRET_MANAGER_NAME>",
      "key": "latest"
    }
  }
}
Can you confirm whether your work pool is the Cloud Run v2 worker? If so, switch to 
valueSource.secretKeyRef.secret/version
as shown above. References: - GCP worker guide - Customize job variables
a
"job_configuration": { "name": "{{ name }}", "labels": "{{ labels }}", "region": "{{ region }}", "command": "{{ command }}", "timeout": "{{ timeout }}", "job_body": { "kind": "Job", "spec": { "template": { "spec": { "template": { "spec": { "containers": [ { "env": [ { "name": "NAME", "value": "armor-app" }, { "name": "VERSION", "value": "0.2.0" }, { "name": "BUILD_TAGS", "value": "[\"v0.2.0\"]" }, { "name": "ENV", "value": "stage" }, { "name": "DEBUG", "value": "false" }, { "name": "GCS_PROJECT_ID", "value": "windy-furnace-465017-c3" }, { "name": "DB_DIALECT", "value": "postgresql" }, { "name": "DB_SYNC_DRIVER", "value": "pg8000" }, { "name": "DB_ASYNC_DRIVER", "value": "asyncpg" }, { "name": "DB_USER", "value": "cloud-run-service-account@windy-furnace-465017-c3.iam.gserviceaccount.com" }, { "name": "DB_SUPPORTS_SCHEMA", "value": "true" }, { "name": "DB_NAME", "value": "postgres" }, { "name": "DB_GCP_SQL_INSTANCE_CONNECTION_NAME", "value": "windy-furnace-465017-c3us east1armor-adk-db" }, { "name": "PREFECT_API_URL", "valueFrom": { "secretKeyRef": { "key": "latest", "name": "PREFECT_API_URL" } } }, { "name": "PREFECT_SERVER_API_AUTH_STRING", "valueFrom": { "secretKeyRef": { "key": "latest", "name": "PREFECT_SERVER_API_AUTH_STRING" } } }, { "name": "PREFECT_API_AUTH_STRING", "valueFrom": { "secretKeyRef": { "key": "latest", "name": "PREFECT_API_AUTH_STRING" } } }, { "name": "LANGFUSE_SECRET_KEY", "valueFrom": { "secretKeyRef": { "key": "latest", "name": "LANGFUSE_SECRET_KEY" } } }, { "name": "LANGFUSE_PUBLIC_KEY", "valueFrom": { "secretKeyRef": { "key": "latest", "name": "LANGFUSE_PUBLIC_KEY" } } }, { "name": "LANGFUSE_HOST", "valueFrom": { "secretKeyRef": { "key": "latest", "name": "LANGFUSE_HOST" } } }, { "name": "GH_TOKEN", "valueFrom": { "secretKeyRef": { "key": "latest", "name": "GH_TOKEN" } } }, { "name": "GOOGLE_API_KEY", "valueFrom": { "secretKeyRef": { "key": "latest", "name": "GOOGLE_API_KEY" } } }, { "name": "GOOGLE_SEARCH_CX", "valueFrom": { "secretKeyRef": { "key": "latest", "name": "GOOGLE_SEARCH_CX" } } } ], "image": "{{ image }}", "command": "{{ command }}", "resources": { "limits": { "cpu": "{{ cpu }}", "memory": "{{ memory }}" }, "requests": { "cpu": "{{ cpu }}", "memory": "{{ memory }}" } } } ], "timeoutSeconds": "{{ timeout }}", "serviceAccountName": "{{ service_account_name }}" } } }, "metadata": { "annotations": { "run.googleapis.com/cloudsql-instances": "windy-furnace-465017-c3us east1armor-adk-db", "run.googleapis.com/vpc-access-connector": "{{ vpc_connector_name }}" } } } }, "metadata": { "name": "{{ name }}", "annotations": { "run.googleapis.com/launch-stage": "BETA" } }, "apiVersion": "run.googleapis.com/v1" }, "keep_job": "{{ keep_job }}", "credentials": "{{ credentials }}" } this is my job_configuration, using the v1, even tho i have env vars, when i inspect the cloud run job the env vars are not populated @Marvin
m
✅ thought for 180.2 seconds
n
@Amith M you can ask marvin to create it for you, but i would really appreciate if you could create a discussion on this so that others can benefit from your exploration here. seen questions come up around this in the past, discussions are a great way to promote off-hand questions onto our radar so they can eventually go in the docs
a
@Nate cool will do this
m
Thanks for sharing the template — a couple things jump out that explain why your env vars aren’t showing up on the created Cloud Run Job. Key points: 1) Your YAML/JSON nesting for Cloud Run Job v1 is off by one level - For Cloud Run Jobs v1, env should live under spec.template.template.spec.containers[].env - Your snippet has spec.template.spec.template.spec which inserts an extra spec and causes Cloud Run to ignore that containers block 2) The Cloud Run v1 worker may overwrite container.env from job_variables.env - In Prefect 3, workers commonly populate container fields from job_variables; if job_variables.env is present (even empty), it can replace env in your job_body - Safest approach is to put env in job_variables and let the base_job_template point to it What to change A. Fix the v1 Job structure (double “template”, no extra “spec” in between). For example:
Copy code
"job_body": {
  "apiVersion": "<http://run.googleapis.com/v1|run.googleapis.com/v1>",
  "kind": "Job",
  "metadata": {
    "name": "{{ name }}",
    "annotations": {
      "<http://run.googleapis.com/launch-stage|run.googleapis.com/launch-stage>": "BETA"
    }
  },
  "spec": {
    "template": {
      "metadata": {
        "annotations": {
          "<http://run.googleapis.com/cloudsql-instances|run.googleapis.com/cloudsql-instances>": "windy-furnace-465017-c3:us-east1:armor-adk-db",
          "<http://run.googleapis.com/vpc-access-connector|run.googleapis.com/vpc-access-connector>": "{{ vpc_connector_name }}"
        }
      },
      "template": {
        "spec": {
          "containers": [
            {
              "image": "{{ image }}",
              "command": "{{ command }}",
              "env": "{{ env }}",  // see B below
              "resources": {
                "limits": {"cpu": "{{ cpu }}", "memory": "{{ memory }}"},
                "requests": {"cpu": "{{ cpu }}", "memory": "{{ memory }}"}
              }
            }
          ],
          "serviceAccountName": "{{ service_account_name }}",
          "timeoutSeconds": "{{ timeout }}"
        }
      }
    }
  }
}
Notes: - The “env” line uses a template variable so Prefect can inject env from job_variables. - In Cloud Run v1, secret-based env uses valueFrom.secretKeyRef with name/key (your use of key: "latest" is correct). B. Put env vars into job_variables (deployment or pool defaults) Example job_variables you can set on the deployment or in the pool’s defaults: ``` { "env": [ { "name": "NAME", "value": "armor-app" }, { "name": "VERSION", "value": "0.2.0" }, { "name": "BUILD_TAGS", "value": "[\"v0.2.0\"]" }, { "name": "ENV", "value": "stage" }, { "name": "DEBUG", "value": "false" }, { "name": "GCS_PROJECT_ID", "value": "windy-furnace-465017-c3" }, { "name": "DB_DIALECT", "value": "postgresql" }, { "name": "DB_SYNC_DRIVER", "value": "pg8000" }, { "name": "DB_ASYNC_DRIVER", "value": "asyncpg" }, { "name": "DB_USER", "value": "cloud-run-service-account@windy-furnace-465017-c3.iam.gserviceaccount.com" }, { "name": "DB_SUPPORTS_SCHEMA", "value": "true" }, { "name": "DB_NAME", "value": "postgres" }, { "name": "DB_GCP_SQL_INSTANCE_CONNECTION_NAME", "value": "windy-furnace-465017-c3us east1armor-adk-db" }, { "name": "PREFECT_API_URL", "valueFrom": { "secretKeyRef": { "name": "PREFECT_API_URL", "key": "latest" } } }, { "name": "PREFECT_SERVER_API_AUTH_STRING", "valueFrom": { "secretKeyRef": { "name": "PREFECT_SERVER_API_AUTH_STRING", "key": "latest" } } }, { "name": "PREFECT_API_AUTH_STRING", "valueFrom": { "secretKeyRef": { "name": "PREFECT_API_AUTH_STRING", "key": "latest" } } }, { "name": "LANGFUSE_SECRET_KEY", "valueFrom": { "secretKeyRef": { "name": "LANGFUSE_SECRET_KEY", "key": "latest" } } }, { "name": "LANGFUSE_PUBLIC_KEY", "valueFrom": { "secretKeyRef": { "name": "LANGFUSE_PUBLIC_KEY", "key": "latest" } } }, { "name": "LANGFUSE_HOST", "valueFrom": { "secretKeyRef": { "name": "LANGFUSE_HOST", "key": "latest" } } }, { "name": "GH_TOKEN",
"valueFrom": { "secretKeyRef": { "name": "GH_TOKEN", "key": "latest" } } }, { "name": "GOOGLE_API_KEY", "valueFrom": { "secretKeyRef": { "name": "GOOGLE_API_KEY", "key": "latest" } } }, { "name": "GOOGLE_SEARCH_CX", "valueFrom": { "secretKeyRef": { "name": "GOOGLE_SEARCH_CX", "key": "latest" } } } ] } ``` C. Sanity checks - Ensure “command” resolves to the type Cloud Run expects (array of strings for entrypoint/command). If you pass a string, Cloud Run may ignore it. - Verify you didn’t set an empty env list in job_variables at the deployment or pool — that would overwrite env in job_body. - If you move to the Cloud Run v2 worker later, switch to: - env[].valueSource.secretKeyRef.secret/version (instead of valueFrom.secretKeyRef.name/key) If you’d like, paste your base_job_template with the corrected nesting and I can double-check it. Also, which place are you setting env right now — only in job_body, or do you also have job_variables.env on the pool/deployment? References: - GCP worker guide - Customize job variables
n
🙏
a
@Nate can you ping the channel here for discussions
n
just to clarify what i meant: • you can go here and make a github discussion yourself • you can say "hey marvin codify this slack thread as a discussion" when you're done exploring/asking questions and it will create the discussion for you based on the thread
lmk if that doesn't answer your question
a
got it
@Marvin spec.template.spec.template.spec is correct, and spec.template.template.spec does not even deploy, can you point me to right docs with an example
m
✅ thought for 104.6 seconds
Thanks for the pushback — you’re right that Jobs and Services have different nesting. For Cloud Run Jobs (apiVersion run.googleapis.com/v1, kind: Job), the official schema places containers under spec.template.template.spec (double “template”). That’s the structure Google documents for Jobs; Services use spec.template.spec. Official references and examples: - Cloud Run YAML reference (Job v1): YAML: Job (v1) - Cloud Run REST reference (Job v1): REST: projects.locations.jobs - Using Secret Manager with env vars (works for Jobs too): Configure secrets as environment variables Minimal example (Job v1) showing env and Secret Manager refs:
Copy code
{
  "apiVersion": "<http://run.googleapis.com/v1|run.googleapis.com/v1>",
  "kind": "Job",
  "metadata": {
    "name": "example-job"
  },
  "spec": {
    "template": {
      "template": {
        "spec": {
          "containers": [
            {
              "image": "<http://gcr.io/PROJECT/IMAGE:TAG|gcr.io/PROJECT/IMAGE:TAG>",
              "env": [
                {
                  "name": "PLAIN_VAR",
                  "value": "value"
                },
                {
                  "name": "PREFECT_API_URL",
                  "valueFrom": {
                    "secretKeyRef": {
                      "name": "PREFECT_API_URL",
                      "key": "latest"
                    }
                  }
                }
              ]
            }
          ],
          "serviceAccountName": "<mailto:SERVICE-ACCOUNT@PROJECT.iam.gserviceaccount.com|SERVICE-ACCOUNT@PROJECT.iam.gserviceaccount.com>",
          "timeoutSeconds": 3600,
          "restartPolicy": "Never"
        }
      }
    }
  }
}
Notes: - For Job v1, secret-based env uses 
valueFrom.secretKeyRef
with 
name
(Secret Manager secret) and 
key
(version, e.g., "latest"). - If a manifest with 
spec.template.template.spec
“does not deploy,” double-check you’re applying it as a Job (not a Service), and that the API version is 
<http://run.googleapis.com/v1|run.googleapis.com/v1>
with 
kind: Job
. The Service schema is different and will reject the Jobs structure. - In Prefect 3, if you also define 
job_variables.env
, that can overwrite env from your 
job_body
. If env still isn’t present at runtime, verify you don’t set an empty env list at the pool or deployment level. If you want to move to the Cloud Run v2 worker later, switch env refs to:
Copy code
{
  "name": "PREFECT_API_URL",
  "valueSource": {
    "secretKeyRef": {
      "secret": "PREFECT_API_URL",
      "version": "latest"
    }
  }
}
If you share the exact error you hit when deploying the 
spec.template.template.spec
form, I can help pinpoint what Cloud Run is rejecting.
a
@Marvin "job_body": { "apiVersion": "run.googleapis.com/v1", "kind": "Job", "metadata": { "name": "{{ name }}", "annotations": { "run.googleapis.com/launch-stage": "BETA" } }, "spec": { "template": { "spec": { "template": { "spec": { "containers": [ { "image": "{{ image }}", "command": "{{ command }}", "env": [ {"name": "test", "value": "test"}, ], "resources": { "limits": {"cpu": "{{ cpu }}", "memory": "{{ memory }}"}, "requests": {"cpu": "{{ cpu }}", "memory": "{{ memory }}"} } } ], "timeoutSeconds": "{{ timeout }}", "serviceAccountName": "{{ service_account_name }}" } } }, "metadata": { "annotations": { "run.googleapis.com/vpc-access-connector": "{{ vpc_connector_name }}", "run.googleapis.com/cloudsql-instances": _CLOUDSQL_INSTANCE_NAME } } } } } this is the job body i am using, but the yaml in the gcp job generated is: apiVersion: run.googleapis.com/v1 kind: Job metadata: name: stage-cloudrun-job namespace: '675776943808' selfLink: /apis/run.googleapis.com/v1/namespaces/675776943808/jobs/stage-cloudrun-job uid: 241afcb9-7379-431d-9d2b-dbbde6f0a5d2 resourceVersion: AAZBrpfE4vk generation: 1 creationTimestamp: '2025-10-21T173300.947177Z' labels: cloud.googleapis.com/location: us-east1 run.googleapis.com/lastUpdatedTime: '2025-10-21T173300.947177Z' annotations: run.googleapis.com/creator: amithmundur@gmail.com run.googleapis.com/lastModifier: amithmundur@gmail.com run.googleapis.com/operation-id: 6b7d891c-f236-4f72-88b8-eafb88e78b14 spec: template: metadata: annotations: run.googleapis.com/cloudsql-instances: windy-furnace-465017-c3us east1armor-adk-db run.googleapis.com/execution-environment: gen2 spec: taskCount: 1 template: spec: containers: - image: gcr.io/windy-furnace-465017-c3/armor-prefect-worker-stage:1b374e7 command: - prefect - flow-run - execute env: - name: PREFECT_API_URL value: http://34.74.70.218:4200/api - name: PREFECT_API_AUTH_STRING value: admin:hufflepuff - name: PREFECT_SERVER_API_AUTH_STRING value: admin:hufflepuff - name: PREFECT__FLOW_RUN_ID value: 94f40e2e-e1a3-4d47-b04e-82b17e8e56b5 resources: limits: cpu: 2000m memory: 4Gi requests: cpu: 2000m memory: 4Gi maxRetries: 3 timeoutSeconds: '600' serviceAccountName: cloud-run-service-account@windy-furnace-465017-c3.iam.gserviceaccount.com status: observedGeneration: 1 conditions: - type: Ready status: 'True' lastTransitionTime: '2025-10-21T173301.540408Z' executionCount: 1 latestCreatedExecution: name: stage-cloudrun-job-ncsm2 creationTimestamp: '2025-10-21T173303.280629Z' completionStatus: EXECUTION_RUNNING the envs are different to what i have mentioned and looks like its being changed by prefect
m
✅ thought for 29.9 seconds
You’re seeing this because the Cloud Run worker overwrites container.env when it builds the job request. In Prefect 3, the Cloud Run (v1) worker injects its own required env (e.g., PREFECT_API_URL, PREFECT__FLOW_RUN_ID, etc.) and uses job_variables.env as the source of any additional env you want. An env array hard-coded inside job_body will be ignored/overwritten. What to do: - Put your env vars (including Secret Manager refs) in job_variables.env at either: - the work pool defaults, or - the deployment’s job_variables Example job_variables.env (Cloud Run v1 uses valueFrom.secretKeyRef name/key):
Copy code
{
  "env": [
    { "name": "test", "value": "test" },

    { "name": "PREFECT_API_URL",
      "valueFrom": { "secretKeyRef": { "name": "PREFECT_API_URL", "key": "latest" } }
    },
    { "name": "PREFECT_SERVER_API_AUTH_STRING",
      "valueFrom": { "secretKeyRef": { "name": "PREFECT_SERVER_API_AUTH_STRING", "key": "latest" } }
    },
    { "name": "PREFECT_API_AUTH_STRING",
      "valueFrom": { "secretKeyRef": { "name": "PREFECT_API_AUTH_STRING", "key": "latest" } }
    }
    // ...your other envs
  ]
}
Then in your base_job_template, either omit env from job_body or set it to the template variable so Prefect can merge/inject:
Copy code
"env": "{{ env }}"
A few notes: - Your Job v1 nesting is fine: spec.template.template.spec.containers[].env is the correct spot for Jobs. - Prefect will also override command/image/resources/etc. from job_variables and required defaults. If you need to change those, set them via job_variables (not by hard-coding in job_body). - If you later switch to the Cloud Run v2 worker, change secret refs to valueSource.secretKeyRef with secret/version. Docs: - Customize job variables: https://docs-3.prefect.io/v3/how-to-guides/deployments/customize-job-variables - GCP worker guide: https://docs-3.prefect.io/integrations/prefect-gcp/gcp-worker-guide If you share how you’re setting job_variables today (pool defaults vs deployment), I can suggest the exact place/snippet to add your env list.
a
@Marvin am close now:
Copy code
container_env_vars = {
    "NAME": "armor-app",
    "VERSION": "0.2.0",
    "BUILD_TAGS": '["v0.2.0"]',
    "ENV": env,
    "DEBUG": "false",
    "GCS_PROJECT_ID": _PROJECT_ID,
    "DB_DIALECT": "postgresql",
    "DB_SYNC_DRIVER": "pg8000",
    "DB_ASYNC_DRIVER": "asyncpg",
    "DB_USER": _SERVICE_ACCOUNT,
    "DB_SUPPORTS_SCHEMA": "true",
    "DB_NAME": "postgres",
    "DB_GCP_SQL_INSTANCE_CONNECTION_NAME": _CLOUDSQL_INSTANCE_NAME,
}

secret_client = secretmanager.SecretManagerServiceAsyncClient()
print("Checking for environment-specific secrets...")
for secret_base_name in secrets_to_map:
    env_secret_name = f"{secret_base_name}_{env}"
    try:
        # Construct the full resource name of the secret to check for its existence
        secret_path = secret_client.secret_path(_PROJECT_ID, env_secret_name)
        await secret_client.get_secret(request={"name": secret_path})
        print(f"✅ Found environment-specific secret: {env_secret_name}")
        final_secret_name = env_secret_name
    except (NotFound, PermissionDenied):
        print(f"⚠️  '{env_secret_name}' not found. Falling back to default: {secret_base_name}")
        final_secret_name = secret_base_name

    container_env_vars[secret_base_name] = {
        "valueFrom": {
            "secretKeyRef": {
                "name": final_secret_name,
                "key": "latest"
            }
        }
    }
used the above and got: env: - name: PREFECT_API_URL value: '{''valueFrom'': {''secretKeyRef'': {''key'': ''latest'', ''name'': ''PREFECT_API_URL''}}}' - name: PREFECT_API_AUTH_STRING value: '{''valueFrom'': {''secretKeyRef'': {''key'': ''latest'', ''name'': ''PREFECT_API_AUTH_STRING''}}}' - name: PREFECT_SERVER_API_AUTH_STRING value: '{''valueFrom'': {''secretKeyRef'': {''key'': ''latest'', ''name'': ''PREFECT_SERVER_API_AUTH_STRING''}}}' - name: PREFECT__FLOW_RUN_ID value: 94f40e2e-e1a3-4d47-b04e-82b17e8e56b5 - name: ENV value: stage - name: NAME value: armor-app - name: DEBUG value: 'false' - name: DB_NAME value: postgres - name: DB_USER value: cloud-run-service-account@windy-furnace-465017-c3.iam.gserviceaccount.com all the normal ones work as expected but the secret based env vars are not working correctly
m
✅ thought for 48.5 seconds
You’re super close — the issue is the shape you’re passing into 
job_variables.env
. Right now you’re building a dict like:
Copy code
{
  "PREFECT_API_URL": { "valueFrom": {...} },
  "NAME": "armor-app",
  ...
}
When Prefect builds the Cloud Run Job, it expects 
env
to be a list of objects. Since you passed a dict, the worker converts it to 
[{"name": k, "value": str(v)}]
, which stringifies your 
valueFrom
dict. That’s why you see:
Copy code
- name: PREFECT_API_URL
  value: '{''valueFrom'': { ... }}'
Fix: build a list of env objects. Cloud Run v1 uses 
<http://valueFrom.secretKeyRef.name/key|valueFrom.secretKeyRef.name/key>
, so do this: - Build 
env_items
as a list (not a dict) - For secrets, use 
{"name": "<ENV>", "valueFrom": {"secretKeyRef": {"name": "<SECRET>", "key": "latest"}}}
Example adapting your code:
Copy code
container_env_vars = {
    "NAME": "armor-app",
    "VERSION": "0.2.0",
    "BUILD_TAGS": '["v0.2.0"]',
    "ENV": env,
    "DEBUG": "false",
    "GCS_PROJECT_ID": _PROJECT_ID,
    "DB_DIALECT": "postgresql",
    "DB_SYNC_DRIVER": "pg8000",
    "DB_ASYNC_DRIVER": "asyncpg",
    "DB_USER": _SERVICE_ACCOUNT,
    "DB_SUPPORTS_SCHEMA": "true",
    "DB_NAME": "postgres",
    "DB_GCP_SQL_INSTANCE_CONNECTION_NAME": _CLOUDSQL_INSTANCE_NAME,
}

# Add secret-backed envs by name
secret_client = secretmanager.SecretManagerServiceAsyncClient()
for secret_base_name in secrets_to_map:
    env_secret_name = f"{secret_base_name}_{env}"
    try:
        secret_path = secret_client.secret_path(_PROJECT_ID, env_secret_name)
        await secret_client.get_secret(request={"name": secret_path})
        final_secret_name = env_secret_name
    except (NotFound, PermissionDenied):
        final_secret_name = secret_base_name

    container_env_vars[secret_base_name] = {
        "valueFrom": {
            "secretKeyRef": {
                "name": final_secret_name,
                "key": "latest",
            }
        }
    }

# Convert to the list shape Prefect/Cloud Run expects
env_items = []
for name, val in container_env_vars.items():
    if isinstance(val, dict) and "valueFrom" in val:
        env_items.append({"name": name, "valueFrom": val["valueFrom"]})
    else:
        env_items.append({"name": name, "value": str(val)})

# Pass as job_variables to Prefect (deployment or pool defaults)
job_variables = {"env": env_items}
Important notes: - Avoid overriding Prefect’s own system envs (
PREFECT_API_URL
, 
PREFECT_API_AUTH_STRING
, etc.). The worker injects these and may overwrite your values. If you want those to come from Secret Manager, you can: - Put new names in Secret Manager (e.g., 
MY_PREFECT_API_URL
) and set your own code to read them; or - Try the same names, but be aware the worker may still set them from config (last write wins is not guaranteed). - Your Job v1 nesting is fine: 
spec.template.template.spec.containers[].env
. References: - Customize job variables: https://docs-3.prefect.io/v3/how-to-guides/deployments/customize-job-variables - Cloud Run secrets as env vars (v1): https://cloud.google.com/run/docs/configuring/secrets#reference-secret-env If you share how you’re attaching 
job_variables
(deployment vs pool), I can show the exact spot to drop 
env_items
.
a
job_variables = {"env": env_items}. just creates a variable called env in the yaml, i want to have secret based env in the container
@Marvin job_variables = {"env": env_items}. just creates a variable called env in the yaml, i want to have secret based env in the container
m
✅ thought for 96.3 seconds
Got it — the behavior you’re seeing means Prefect is not receiving your env list in the shape it expects, so it’s not mapping it to container.env. It will never create an environment variable literally named "env" if you pass 
job_variables
correctly. Here’s the working pattern for Cloud Run v1: 1) Ensure your base_job_template uses the env template variable In the work pool’s base job template, point container env at the template variable:
Copy code
...
"containers": [
  {
    "image": "{{ image }}",
    "command": "{{ command }}",
    "env": "{{ env }}",     // important: let Prefect inject job_variables.env here
    "resources": {
      "limits": {"cpu": "{{ cpu }}", "memory": "{{ memory }}"},
      "requests": {"cpu": "{{ cpu }}", "memory": "{{ memory }}"}
    }
  }
],
...
Note: For Cloud Run Jobs v1, the nesting is 
spec.template.template.spec.containers[].env
. Keep your double “template” and put the line above there. 2) Pass env as a list of env objects in job_variables 
job_variables.env
must be a list, not a dict. Each item is 
{ "name": ..., "value": ... }
or 
{ "name": ..., "valueFrom": { "secretKeyRef": ... } }
. Example:
Copy code
job_variables = {
  "env": [
    { "name": "NAME", "value": "armor-app" },
    { "name": "ENV", "value": "stage" },

    { "name": "PREFECT_API_URL",
      "valueFrom": { "secretKeyRef": { "name": "PREFECT_API_URL", "key": "latest" } }
    },
    { "name": "PREFECT_API_AUTH_STRING",
      "valueFrom": { "secretKeyRef": { "name": "PREFECT_API_AUTH_STRING", "key": "latest" } }
    },
    { "name": "PREFECT_SERVER_API_AUTH_STRING",
      "valueFrom": { "secretKeyRef": { "name": "PREFECT_SERVER_API_AUTH_STRING", "key": "latest" } }
    },

    { "name": "GOOGLE_API_KEY",
      "valueFrom": { "secretKeyRef": { "name": "GOOGLE_API_KEY", "key": "latest" } }
    }
    // ...add the rest
  ]
}
3) Attach job_variables at deployment or pool defaults - In Python (Prefect 3.x):
Copy code
flow.from_source("<https://github.com/org/repo>", entrypoint="path:flow.py:flow").deploy(
  name="my-deployment",
  work_pool_name="my-cloud-run-pool",
  image="<your-image>",
  job_variables=job_variables,
)
- Or in prefect.yaml under your deployment:
Copy code
deployments:
  - name: my-deployment
    work_pool:
      name: my-cloud-run-pool
    job_variables:
      env:
        - name: NAME
          value: armor-app
        - name: PREFECT_API_URL
          valueFrom:
            secretKeyRef:
              name: PREFECT_API_URL
              key: latest
        # ...
Why your secret envs were stringified When you previously built a dict like:
Copy code
{"PREFECT_API_URL": {"valueFrom": { ... }}}
Prefect treated it as a single env var with name "PREFECT_API_URL" and value equal to the stringified dict, because it expects a list of objects, not a dict mapping. That’s why you saw:
Copy code
- name: PREFECT_API_URL
  value: '{''valueFrom'': {...}}'
Final tips - Keep your Job v1 nesting: spec.template.template.spec.containers[].env is correct for Jobs. - Don’t hard-code an env array in job_body; it will be overwritten. Use the template hook 
"env": "{{ env }}"
. - For Cloud Run v1, secret references use 
<http://valueFrom.secretKeyRef.name/key|valueFrom.secretKeyRef.name/key>
. - If you ever move to Cloud Run v2 worker, switch to 
valueSource.secretKeyRef.secret/version
. Docs: - Customize job variables: https://docs-3.prefect.io/v3/how-to-guides/deployments/customize-job-variables - GCP worker guide: https://docs-3.prefect.io/integrations/prefect-gcp/gcp-worker-guide - Cloud Run secrets as env vars (v1): https://cloud.google.com/run/docs/configuring/secrets#reference-secret-env If you share how you’re injecting job_variables (Python deploy vs prefect.yaml) and your current base_job_template snippet for the container, I can validate it end-to-end.
a
@Marvin i am doing this in variables:
Copy code
"env": {
    "title": "Environment Variables",
    "description": "Environment variables to set when starting a flow run.", "type": "object",
    "additionalProperties": {"type": "string"},
    "default": [
        {"name": "test", "value": "test"},
        {"name": "env", "valueFrom": {
            "secretKeyRef": {
                "name": "test",
                "key": "latest"
            }
        }},
    ],
}
i get: File "/Users/amithmundur/PycharmProjects/armor-adk/.venv/lib/python3.12/site-packages/pydantic/main.py", line 253, in init validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/amithmundur/PycharmProjects/armor-adk/.venv/lib/python3.12/site-packages/prefect/workers/base.py", line 146, in _coerce_env return {k: str(v) if v is not None else None for k, v in v.items()} ^^^^^^^ AttributeError: 'list' object has no attribute 'items'
m
✅ thought for 39.3 seconds
You’re hitting a naming/type collision with the built-in env variable on BaseVariables. - Prefect’s BaseVariables defines env as a mapping of string->string. When you set env in the work pool variables as a list, Pydantic tries to coerce it to a dict and fails with v.items() (your error). - That env is for the worker’s process environment, not the container’s Cloud Run env. How to set secret-based env vars on the Cloud Run container 1) Use a different variable name for the container env list Define a new variable (e.g., container_env) in the work pool’s base_job_template variables schema, typed as an array of objects. Then reference it in the job_body at containers[].env. Example base_job_template snippet:
Copy code
{
  "variables": {
    "type": "object",
    "properties": {
      "container_env": {
        "title": "Container environment variables",
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "name": {
              "type": "string"
            },
            "value": {
              "type": [
                "string",
                "null"
              ]
            },
            "valueFrom": {
              "type": "object",
              "properties": {
                "secretKeyRef": {
                  "type": "object",
                  "properties": {
                    "name": {
                      "type": "string"
                    },
                    "key": {
                      "type": "string"
                    }
                  },
                  "required": [
                    "name",
                    "key"
                  ]
                }
              }
            }
          },
          "required": [
            "name"
          ]
        },
        "default": []
      },
      "image": {
        "type": "string"
      },
      "command": {
        "type": [
          "array",
          "null"
        ],
        "items": {
          "type": "string"
        }
      },
      "cpu": {
        "type": "string"
      },
      "memory": {
        "type": "string"
      },
      "timeout": {
        "type": "integer"
      },
      "service_account_name": {
        "type": "string"
      },
      "vpc_connector_name": {
        "type": "string"
      }
    },
    "required": [
      "image"
    ]
  },
  "job_configuration": {
    "job_body": {
      "apiVersion": "<http://run.googleapis.com/v1|run.googleapis.com/v1>",
      "kind": "Job",
      "metadata": {
        "name": "{{ name }}"
      },
      "spec": {
        "template": {
          "metadata": {
            "annotations": {
              "<http://run.googleapis.com/vpc-access-connector|run.googleapis.com/vpc-access-connector>": "{{ vpc_connector_name }}"
            }
          },
          "template": {
            "spec": {
              "containers": [
                {
                  "image": "{{ image }}",
                  "command": "{{ command }}",
                  "env": "{{ container_env }}",
                  "resources": {
                    "limits": {
                      "cpu": "{{ cpu }}",
                      "memory": "{{ memory }}"
                    },
                    "requests": {
                      "cpu": "{{ cpu }}",
                      "memory": "{{ memory }}"
                    }
                  }
                }
              ],
              "serviceAccountName": "{{ service_account_name }}",
              "timeoutSeconds": "{{ timeout }}"
            }
          }
        }
      }
    }
  }
}
2) Provide container_env via job_variables at deployment (or pool defaults) Cloud Run v1 secret refs use valueFrom.secretKeyRef.name/key.
Copy code
job_variables = {
  "container_env": [
    { "name": "NAME", "value": "armor-app" },
    { "name": "ENV", "value": "stage" },

    { "name": "GOOGLE_API_KEY",
      "valueFrom": { "secretKeyRef": { "name": "GOOGLE_API_KEY", "key": "latest" } }
    },
    { "name": "PREFECT_API_URL",
      "valueFrom": { "secretKeyRef": { "name": "PREFECT_API_URL", "key": "latest" } }
    }
    // ...more
  ]
}
3) Don’t use the reserved env in variables for container env - Keep BaseVariables.env (dict) for simple worker-level env if you need it. - Use container_env (array) for Cloud Run container env. This avoids the Pydantic coercion error and allows secret refs to be passed through as objects. Notes - Prefect will still inject its own required env into the container (e.g., PREFECT_API_URL, PREFECT__FLOW_RUN_ID). Your container_env will be merged in addition to those.
- For Cloud Run v2, the secret syntax changes to valueSource.secretKeyRef.secret/version. - Make sure command is an array of strings for Cloud Run. Refs: - Customize job variables: https://docs-3.prefect.io/v3/how-to-guides/deployments/customize-job-variables - GCP worker guide: https://docs-3.prefect.io/integrations/prefect-gcp/gcp-worker-guide - Cloud Run secrets as env vars: https://cloud.google.com/run/docs/configuring/secrets#reference-secret-env If you paste your full base_job_template variables block and the containers[].env line, I can sanity-check it for you.
a
@Marvin but i cant directly use "env" in container since its being overwritten and only: env: - name: PREFECT_API_URL value: http://34.74.70.218:4200/api - name: PREFECT_API_AUTH_STRING value: admin:hufflepuff - name: PREFECT_SERVER_API_AUTH_STRING value: admin:hufflepuff - name: PREFECT__FLOW_RUN_ID value: 94f40e2e-e1a3-4d47-b04e-82b17e8e56b5 - name: PYTHONPATH value: src/app these are filled
m
✅ thought for 92.6 seconds
You’ve run into a schema mismatch: the Cloud Run worker’s built-in variables define env as a dict (string->string), so Prefect will not pass through valueFrom objects, and it also doesn’t wire that dict into containers[].env by default. That’s why only Prefect’s own system env show up. Use a dedicated variable for container env (an array of objects) and map it directly into containers[].env in your base job template. This bypasses the dict-based env on BaseVariables and lets you include valueFrom.secretKeyRef. Do this: 1) Export and edit your pool’s base job template - Get the template:
Copy code
prefect work-pool get-default-base-job-template --type cloud-run --file base_job_template.json
- Edit base_job_template.json: - Add a new variables property container_env (array of objects) - Map containers[].env to "{{ container_env }}" Example minimal patch:
Copy code
{
  "job_configuration": {
    ...
    "job_body": {
      "apiVersion": "<http://run.googleapis.com/v1|run.googleapis.com/v1>",
      "kind": "Job",
      "spec": {
        "template": {
          "spec": {
            "template": {
              "spec": {
                "containers": [
                  {
                    "image": "{{ image }}",
                    "command": "{{ command }}",
                    "env": "{{ container_env }}",   // <-- wire your array here
                    "resources": {
                      "limits": { "cpu": "{{ cpu }}", "memory": "{{ memory }}" },
                      "requests": { "cpu": "{{ cpu }}", "memory": "{{ memory }}" }
                    }
                  }
                ],
                "timeoutSeconds": "{{ timeout }}",
                "serviceAccountName": "{{ service_account_name }}"
              }
            }
          },
          "metadata": {
            "annotations": {
              "<http://run.googleapis.com/vpc-access-connector|run.googleapis.com/vpc-access-connector>": "{{ vpc_connector_name }}"
            }
          }
        }
      }
    },
    ...
  },
  "variables": {
    "type": "object",
    "properties": {
      ...
      "container_env": {
        "title": "Container environment variables",
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "name": { "type": "string" },
            "value": { "type": ["string", "null"] },
            "valueFrom": {
              "type": "object",
              "properties": {
                "secretKeyRef": {
                  "type": "object",
                  "properties": {
                    "name": { "type": "string" },
                    "key": { "type": "string" }
                  },
                  "required": ["name", "key"]
                }
              }
            }
          },
          "required": ["name"]
        },
        "default": []
      }
    }
  }
}
- Update the pool with your edited template:
Copy code
prefect work-pool update "<your-pool>" --base-job-template base_job_template.json
2) Provide your secret-based env as an array via job_variables.container_env Cloud Run v1 secret refs use valueFrom.secretKeyRef.name/key.
Copy code
job_variables = {
  "container_env": [
    { "name": "NAME", "value": "armor-app" },
    { "name": "ENV", "value": "stage" },

    { "name": "PREFECT_API_URL",
      "valueFrom": { "secretKeyRef": { "name": "PREFECT_API_URL", "key": "latest" } }
    },
    { "name": "PREFECT_API_AUTH_STRING",
      "valueFrom": { "secretKeyRef": { "name": "PREFECT_API_AUTH_STRING", "key": "latest" } }
    },
    { "name": "PREFECT_SERVER_API_AUTH_STRING",
      "valueFrom": { "secretKeyRef": { "name": "PREFECT_SERVER_API_AUTH_STRING", "key": "latest" } }
    },
    { "name": "GOOGLE_API_KEY",
      "valueFrom": { "secretKeyRef": { "name": "GOOGLE_API_KEY", "key": "latest" } }
    }
    // add the rest...
  ]
}
Then deploy with: ``` flow.from_source("https://github.com/org/repo", entrypoint="path/to:flow").deploy( name="my-deployment", work_pool_name="<your-pool>", image="<your-image>",
job_variables=job_variables, ) ``` Notes: - Prefect will still inject its own required env (PREFECT_API_URL, PREFECT__FLOW_RUN_ID, etc.). Your container_env is merged alongside them. - Avoid redefining Prefect’s own env names if you don’t want them overwritten. - Your Cloud Run Job v1 nesting spec.template.template.spec.containers[].env is correct. Docs: - Customize job variables: https://docs-3.prefect.io/v3/how-to-guides/deployments/customize-job-variables - GCP worker guide: https://docs-3.prefect.io/integrations/prefect-gcp/gcp-worker-guide - Cloud Run secrets as env vars: https://cloud.google.com/run/docs/configuring/secrets#reference-secret-env If you paste your current base_job_template (variables + containers block), I can spot-check it for any schema typos.
a
@Marvin YES even if i hardcode the "env" in container, it jets ignored
m
✅ thought for 37.6 seconds
You’re running into a limitation of the Cloud Run v1 worker: it overwrites the container’s env when it builds the Job. The v1 worker only supports 
variables.env
as a dict[str, str] and does not pass through 
valueFrom
objects. As a result: - Any 
containers[].env
you hardcode in 
job_body
is ignored/overwritten - 
variables.env
cannot carry 
valueFrom.secretKeyRef
because it’s coerced to a dict of strings That’s why you only see the Prefect-injected envs (PREFECT_API_URL, PREFECT__FLOW_RUN_ID, etc.) in the resulting Job. Your options: Option 1 — Switch to the Cloud Run v2 worker (recommended) - v2 is more flexible and you can wire an array of env objects directly to the container. Steps: 1) Create a v2 pool and export its template
Copy code
prefect work-pool create my-cr-v2 --type cloud-run-v2
prefect work-pool get-default-base-job-template --type cloud-run-v2 --file base_v2.json
2) Edit base_v2.json: add a new variable 
container_env
(array of objects) and map it to the container’s env: - Add this under variables.properties:
Copy code
"container_env": {
  "title": "Container environment variables",
  "type": "array",
  "items": {
    "type": "object",
    "properties": {
      "name": { "type": "string" },
      "value": { "type": ["string", "null"] },
      "valueSource": {
        "type": "object",
        "properties": {
          "secretKeyRef": {
            "type": "object",
            "properties": {
              "secret": { "type": "string" },
              "version": { "type": "string" }
            },
            "required": ["secret", "version"]
          }
        }
      }
    },
    "required": ["name"]
  },
  "default": []
}
- Map it into job_body at:
Copy code
job_body.template.template.containers[0].env = "{{ container_env }}"
3) Update the pool with your edited template
Copy code
prefect work-pool update my-cr-v2 --base-job-template base_v2.json
4) Provide your secrets in deployment/job_variables using v2 syntax:
Copy code
job_variables = {
  "container_env": [
    { "name": "NAME", "value": "armor-app" },
    {
      "name": "PREFECT_API_URL",
      "valueSource": {
        "secretKeyRef": { "secret": "PREFECT_API_URL", "version": "latest" }
      }
    }
    // …
  ]
}
Option 2 — Stay on Cloud Run v1 but use a custom worker or fetch secrets at runtime - Custom worker: Subclass the Cloud Run v1 worker to accept a 
container_env
array and merge it into 
containers[].env
before submit. This bypasses the dict[str, str] coercion. - Fetch at runtime: Read from Secret Manager in your flow or entrypoint and export to 
os.environ
before running your app. Why hardcoding containers[].env in v1 job_body is ignored - The v1 worker reconstructs the container spec and sets env from its internal logic (Prefect-required vars + variables.env dict). It doesn’t merge or respect a pre-populated 
job_body.containers[].env
. Docs for reference: - GCP worker guide: https://docs-3.prefect.io/integrations/prefect-gcp/gcp-worker-guide - Customize job variables: https://docs-3.prefect.io/v3/how-to-guides/deployments/customize-job-variables - Cloud Run secrets as env vars: https://cloud.google.com/run/docs/configuring/secrets#reference-secret-env If you’re open to v2, I can share a ready-to-paste base_v2.json patch with 
container_env
wired correctly. If you must stay on v1, I can sketch a minimal custom worker that preserves 
valueFrom.secretKeyRef
into the container.
a
@Marvin yup v2 solved the issue
m
✅ thought for 9.9 seconds
Awesome — glad switching to Cloud Run v2 did the trick. Quick notes for v2 so you’ve got the pattern handy: - Secret refs use valueSource.secretKeyRef with secret/version. - Example env item:
Copy code
{
  "name": "PREFECT_API_URL",
  "valueSource": {
    "secretKeyRef": {
      "secret": "PREFECT_API_URL",
      "version": "latest"
    }
  }
}
- Make sure the service account on the Job has Secret Manager Secret Accessor (roles/secretmanager.secretAccessor) to read those secrets. If you haven’t already, a good starting point is exporting the default v2 template and patching it:
Copy code
prefect work-pool get-default-base-job-template --type cloud-run-v2 --file base_v2.json
Then wire your container env: - job_body.template.template.containers[0].env = "{{ container_env }}" - variables.properties.container_env = array of objects, each with name/value or name/valueSource Refs: - GCP worker guide: https://docs-3.prefect.io/integrations/prefect-gcp/gcp-worker-guide - Customize job variables: https://docs-3.prefect.io/v3/how-to-guides/deployments/customize-job-variables If you want, share your final base_v2.json and I can sanity-check it.
a
@Marvin convert this thread into gh discussion
m
✅ thought for 25.5 seconds
Done! I created a GitHub discussion summarizing the problem and solution: Cloud Run work pool: Secret-based env vars ignored in v1; solution is Cloud Run v2 with valueSource.secretKeyRef https://github.com/PrefectHQ/prefect/discussions/19248 If you want me to add any code snippets, logs, or your final base_v2.json for posterity, send them here and I’ll append to the thread.
n
thank you @Amith M
❤️ 1
11 Views
Open in Slack
PreviousNext
Powered by