Riya Sinha
09/06/2025, 9:31 PM"env": {
"PYTHONUNBUFFERED": "1",
"PREFECT_LOGGING_LEVEL": "DEBUG",
"GOOGLE_APPLICATION_CREDENTIALS": "{{ prefect.blocks.secret.gcp-service-account-json-cloud }}"
},
however this results in the flow immediately crashing with error:
Flow run failed during submission. Please contact support.
Marvin
09/06/2025, 9:31 PMMarvin
09/06/2025, 9:36 PM{{ prefect.blocks... }}
syntax isn’t supported in the deployment templating context, so it fails during submission.
Why your example fails:
- Deployment/job_variable templating does not expose blocks, so {{ prefect.blocks.secret.gcp-service-account-json-cloud }}
can’t be resolved and the worker crashes at submission with that generic error.
- Separately, GOOGLE_APPLICATION_CREDENTIALS
must be a filesystem path to a JSON file, not the JSON contents. Even if templating worked, pointing the env var to the raw secret string wouldn’t work with GCP libraries.
Recommended ways to wire GCP credentials
Pick one of these patterns:
1) Load the block in code (recommended)
- Don’t rely on env vars; use a creds object directly in your flow.
- With Prefect’s GCP collection:
from prefect import flow
from prefect_gcp.credentials import GcpCredentials
@flow
def my_flow():
gcp = GcpCredentials.load("my-gcp-creds")
# Example: get a client with these creds (see prefect-gcp docs for service-specific helpers)
creds = gcp.get_credentials_from_service_account()
# pass creds to your GCP client constructors here
Docs: Prefect GCP integration and Store secrets with Blocks
2) If you must use GOOGLE_APPLICATION_CREDENTIALS
- Keep the JSON in a Secret or GcpCredentials block.
- At runtime, write it to a temp file and point the env var to that path before importing/creating GCP clients:
import os, tempfile
from prefect import flow
from prefect.blocks.system import Secret # or use GcpCredentials and its service_account_info
@flow
def my_flow():
sa_json = Secret.load("gcp-service-account-json-cloud").get()
with tempfile.NamedTemporaryFile("w", delete=False) as f:
f.write(sa_json)
path = f.name
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = path
# now import/create clients that rely on the env var
3) Mount credentials via your runtime (Docker/Kubernetes/VM)
- Put the JSON file on the filesystem (e.g., Docker secret/volume, Kubernetes Secret mounted as a file).
- Set GOOGLE_APPLICATION_CREDENTIALS
in your deployment/job_variables to the file path (static string). This avoids putting secret values into deployment metadata and relies on your infra to provide the file securely.
Docs: Customize job variables
Notes
- Avoid putting raw secret values into deployment env vars—they’re stored alongside the deployment and could be exposed to teammates with access.
- The “Flow run failed during submission” is consistent with a template rendering error; check your worker logs for a Jinja/templating stack trace.
If you share which worker/work pool you’re using (Process, Docker, Kubernetes) and whether your block holds the JSON contents vs a path, I can give you a tailored snippet for your setup.