Balazs
09/09/2025, 2:58 PMMarvin
09/09/2025, 2:58 PMMarvin
09/09/2025, 3:07 PMprefect work-pool get-default-base-job-template --type cloud-run
You’ll see fields like:
- region
- image
- cpu, memory (applied to the container resources)
- timeout
- service_account_name
- vpc_connector_name
- credentials (Prefect GCP credentials block)
- keep_job
- job_body (full Cloud Run Job spec template; Prefect sends this to GCP)
Example: per-deployment overrides in prefect.yaml
deployments:
- name: my-cloud-run-deployment
entrypoint: ./flows/etl.py:flow
work_pool:
name: my-cloud-run-pool
job_variables:
image: us-docker.pkg.dev/my-proj/prefect/my-image:latest
region: us-central1
cpu: "2"
memory: "4Gi"
timeout: 3600
service_account_name: <mailto:my-flow-sa@my-proj.iam.gserviceaccount.com|my-flow-sa@my-proj.iam.gserviceaccount.com>
vpc_connector_name: projects/my-proj/locations/us-central1/connectors/my-connector
keep_job: false
Notes on “fan-out” and multi-task Cloud Run jobs
- Prefect’s Cloud Run integration assumes a single task per job (taskCount=1). The default base job template does not set taskCount/parallelism and Prefect does not coordinate multiple Cloud Run tasks for the same flow run.
- Even if you add Cloud Run “taskCount” manually in the job_body, Cloud Run would start N identical tasks running the same Prefect flow-run command, which will conflict. In short: do not try to use Cloud Run Job taskCount>1 for a single flow run.
- To parallelize across multiple Cloud Run containers, trigger multiple flow runs (subflows, mapping to independent flow runs, or using the Runner API) rather than fanning out one flow run.
Capacity planning and concurrency
- Prefect does not do container-level capacity planning or cross-container task placement. Each flow run gets its own Cloud Run Job container and executes there.
- You can:
- Limit how many flow runs are allowed to start in a work pool (work pool concurrency). This is your coarse-grained throttle.
- Control in-container parallelism via your task runner (e.g., ThreadPoolTaskRunner max_workers) if you want parallel task execution inside the single container.
- Rely on Cloud Run quotas/limits for regional scaling and cost controls.
Useful CLI you can rely on
- Inspect default Cloud Run template (to see fields you can set):
prefect work-pool get-default-base-job-template --type cloud-run
- Create a work pool (then edit its base job template in the UI or by supplying your own JSON):
prefect work-pool create "my-cloud-run-pool" --type cloud-run
- Set a concurrency limit on the pool:
prefect work-pool set-concurrency-limit "my-cloud-run-pool" 10Marvin
09/09/2025, 3:07 PMBring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by