<@ULVA73B9P> We put a deployment in Prefect Cloud ...
# marvin-ai
m
@Marvin We put a deployment in Prefect Cloud with it's own unique environment variable, but the presence of that single variable apparently erased the environment variables provided by the work pool (Google Cloud Run). Is this expected behavior?
m
thought for 231.0 seconds
Short answer: yes — that’s expected in Prefect 3.x today. Details: - Deployment-level
job_variables
overwrite work pool defaults at the field level. Since
env
is a single field (a dict), specifying
env
on the deployment replaces the pool’s entire
env
mapping rather than merging keys. - This is consistent with the docs language that deployment job variables “overwrite” work pool job template values. What to do instead: - If you want to keep the work pool’s environment variables and add one for a single deployment, either: - Don’t set
env
on the deployment; keep all env vars on the work pool, or - Include all of the pool env vars plus your extra var in the deployment’s
env
(i.e., fully specify the
env
dict at the deployment), or - Move the deployment-specific value into a Secrets/Blocks approach and reference it from your code so you don’t need to duplicate
env
. References: - Customize job variables (work pool vs deployment) - GCP worker guide (Cloud Run) If you’d like, share how you set the deployment
env
(UI vs
prefect.yaml
vs
.deploy
) and I can suggest the cleanest way to preserve the pool envs while adding your one-off variable.
m
@Marvin Doesn't the following Prefect Github issue suggest that dictionary merging is desirable behavior? It also seems intuitive because environment variables meant to configure your infra may be different than those needed to run a deployment; however, you don't necessarily want to exclude one at the expense of the other https://github.com/PrefectHQ/prefect/issues/11041
m
thought for 261.8 seconds
Great find — yes, that issue does argue that deep-merge of job variables (including env dicts) is desirable: - Issue: Job variables should use a deep merge rather than shallow merge of dictionaries However, today that’s not how Prefect behaves. In current Prefect 3.x, deployment-level
job_variables.env
replaces the work pool’s
env
dict instead of merging keys. That’s why adding a single env var on the deployment appears to “erase” the pool’s env — the deployment’s
env
dict completely overrides the pool’s. This aligns with our docs that say deployment job variables “overwrite” pool defaults: - Customize job variables - GCP worker guide (Cloud Run) If deep-merge were implemented, your expected outcome (infra env from the pool + deployment-specific env working together) would “just work.” It’s a reasonable request; it just isn’t the current behavior. Workarounds/patterns you can use now: - Keep infra env exclusively on the work pool and avoid setting
env
on the deployment. Put deployment-specific config in Prefect Blocks/Secrets or parameters so you don’t have to touch the
env
dict at the deployment level. - If you must set
env
on the deployment, include both the pool envs and the deployment-specific keys in the deployment’s
env
(i.e., fully specify the dict). - If infra envs truly differ by deployment, consider separate work pools so you don’t need to merge. Quick question so I can tailor guidance precisely: are you on Prefect 3.x or 2.x? The behavior is effectively the same here, but #11041 was filed against 2.x. If you’d like, I can help file or upvote a 3.x-specific feature request to deep-merge job variables so envs compose intuitively.
m
@Marvin I'm on 3.x. Would filing a new issue be helpful? Or reopening the old one?
m
thought for 23.6 seconds
Short answer: open a new 3.x issue and link to the old one. Why: - #11041 was filed against 2.x. The behavior you’re seeing in 3.x is “working as implemented” (deployment job_variables replace, they don’t deep-merge). A 3.x-specific feature request is the clearest way to track this. - In your new issue, link to #11041 for context and call out the desired deep-merge semantics. Helpful links: - Old issue for context: https://github.com/PrefectHQ/prefect/issues/11041 - Create new issue: https://github.com/PrefectHQ/prefect/issues/new/choose - Docs describing overwrite behavior: https://docs-3.prefect.io/v3/how-to-guides/deployments/customize-job-variables, https://docs-3.prefect.io/integrations/prefect-gcp/gcp-worker-guide What to include in the new issue (you can paste this): - Title: Deep-merge deployment job_variables.env with work pool env in Prefect 3.x - Version: Prefect 3.x (include exact version) - Worker: Cloud Run (or your worker type) - Current behavior:
Copy code
Work pool base job template:
env:
  POOL_ONLY: "yes"
  SHARED: "pool"

Deployment job_variables:
  env:
    DEPLOY_ONLY: "true"
    SHARED: "deploy"

Observed at runtime: only DEPLOY_ONLY + SHARED=deploy. POOL_ONLY is missing.
- Expected behavior: - Deep-merge dicts so keys from the pool env are preserved and deployment keys override matching keys: - POOL_ONLY=yes, DEPLOY_ONLY=true, SHARED=deploy - Merge precedence: flow run > deployment > work pool - Rationale: Infra-level env vars differ from app-level env vars; replacing the whole dict makes composition cumbersome. - Prior art: link to #11041 If you share your pool template and deployment snippet here, I can help refine the issue text before you file it.