Great question. Here’s a concise, battle-tested pattern for Prefect Cloud + dbt Core + BigQuery across dev/staging/prod, with a focus on eliminating key files and keeping anything sensitive masked. High-level best practices - Production: Prefer no key files at all. Use GCP Workload Identity / Application Default Credentials (ADC) so credentials come from the runtime (Cloud Run/GKE metadata server). This avoids long-lived keys entirely. - Non-prod (dev/staging): If you can, also use ADC (e.g.,

gcloud auth application-default login

locally, or Workload Identity on staging infra). If you must use a service account, store the JSON in a Prefect Block (encrypted), not on disk or in env files. - Separate everything by environment: service accounts, IAM roles, Prefect Blocks, and work pools. Use least-privilege IAM per env. - dbt profiles: Generate profiles from Prefect blocks (BigQueryTargetConfigs + GcpCredentials). Avoid hardcoding secrets in

profiles.yml

or using a key file path in containers. How to do it 1) Use Workload Identity / ADC (no keys, recommended) - In production on Cloud Run/GKE, omit explicit credentials and let

GcpCredentials()

pick up ADC:

from prefect_gcp.credentials import GcpCredentials

# Uses ADC automatically (metadata server in Cloud Run/GKE, or local gcloud if present)
GcpCredentials().save("gcp-creds-prod")

- Ensure your Cloud Run/GKE runtime uses a service account with only the required BigQuery roles (e.g., jobUser + dataEditor for the relevant datasets). Links: - GCP Application Default Credentials (ADC) - Prefect GCP integration 2) If you must use a service account JSON, store it in a Prefect Block (not a file) - Prefer

service_account_info

(SecretDict) over

service_account_file

so you don’t have to mount files into containers. - Each env gets its own block.

from prefect.blocks.system import Secret
from prefect_gcp.credentials import GcpCredentials
import json

# Save the raw service account JSON as a Secret (encrypted, redacted in UI/logs)
sa_info = json.loads("""{ ... full service account JSON ... }""")
Secret(value=json.dumps(sa_info)).save("gcp-sa-json-dev")

# Build a GcpCredentials block from that secret (env-scoped)
gcp_creds_dev = GcpCredentials(service_account_info=sa_info, project="my-project-dev")
gcp_creds_dev.save("gcp-creds-dev")

Notes: - Prefect redacts secret fields (Secret/SecretDict) in logs/UI. Don’t print them yourself. Links: - Blocks overview - Store secrets in Prefect 3) Environment separation - Use separate SA, roles, and Prefect Blocks per env:

gcp-creds-dev

,

gcp-creds-staging

,

gcp-creds-prod

. - Use separate work pools per env so runtime identity and resources are isolated. - Optionally, store non-sensitive per-env settings (dataset, location) in Prefect Variables.

# Work pools (example; exact type depends on your infra)
prefect work-pool create --type cloud-run dev-pool
prefect work-pool create --type cloud-run staging-pool
prefect work-pool create --type cloud-run prod-pool

Links: - Work pools - Variables 4) dbt Core + BigQuery profile via Prefect blocks (no plaintext secrets) - Use

prefect-dbt

to generate a BigQuery profile from Prefect blocks. If you’re using ADC/Workload Identity, you can omit key JSON; if you’re using a SA JSON,

keyfile_json

is provided from your

GcpCredentials

block. ``` from prefect_dbt.cli.configs import BigQueryTargetConfigs from prefect_dbt.cli.credentials import DbtCliProfile from prefect_gcp.credentials import GcpCredentials # Load env-specific creds block gcp_creds = GcpCredentials.load("gcp-creds-dev")

# BigQuery target configs reference GcpCredentials (uses ADC or keyfile_json automatically) bq_cfg = BigQueryTargetConfigs( project="my-project-dev", schema="analytics_dev", credentials=gcp_creds ) bq_cfg.save("bq-target-dev") # dbt CLI profile profile = DbtCliProfile( name="my_dbt_project", target="dev", target_configs=bq_cfg ) profile.save("dbt-profile-dev")

Then in your flow:

from prefect import flow from prefect_dbt import PrefectDbtRunner from prefect_dbt.cli.credentials import DbtCliProfile @flow def run_dbt(): profile = DbtCliProfile.load("dbt-profile-dev") runner = PrefectDbtRunner(settings__project_dir="/app/dbt", settings__profiles_dir="/app/.dbt") runner.invoke(["build"])

Links:
- <https://docs.prefect.io/integrations/prefect-dbt|Prefect dbt integration>
- <https://docs.prefect.io/integrations/prefect-gcp|Prefect GCP integration>

5) What to avoid
- Mounting key files into containers and pointing dbt/BigQuery to `GOOGLE_APPLICATION_CREDENTIALS` (brittle, hard to rotate, leaks in filesystem).
- Saving secrets in `profiles.yml`, `.env`, or Git. Always use Prefect Blocks or ADC.

6) Rotation and masking
- With ADC/Workload Identity, tokens are short-lived and rotation is automatic.
- With blocks, rotate by overwriting the block (no code/redeploy needed):

GcpCredentials(...new SA info...).save("gcp-creds-dev", overwrite=True) ``` - Prefect redacts secrets in logs/UI by default. Avoid printing secrets yourself or passing them to subprocesses via CLI args. 7) Optional: use serverless (push) work pools on GCP - Prefect can submit directly to Cloud Run (no long-running worker), making ADC even simpler in prod. - Keep pools/envs separate. Link: - Serverless (push) work pools Quick checklist per environment - Dev: - Prefer ADC via

gcloud auth application-default login

, or a minimal dev SA stored as a Prefect

GcpCredentials

block (SecretDict). - Staging: - Use Workload Identity if possible; otherwise use a separate staging SA in a

GcpCredentials

block. - Prod: - Use Workload Identity (Cloud Run/GKE). Do not store keys. Separate prod work pool and SA with least privilege. If you can share where your flows and dbt are running (Cloud Run worker/push pool, GKE, VMs, or local CI), I can tailor this to your exact setup and provide a minimal working example.