<@ULVA73B9P> how can i use cloudrun workpool and c...
# ask-marvin
h
@Marvin how can i use cloudrun workpool and cloudrun worker through helm chart when self hosting prefect
m
thought for 348.1 seconds
Great question — you can absolutely self-host Prefect with Helm and run flows on Google Cloud Run. Here’s the practical setup and what’s supported in Prefect 3.x. High-level - Self-host Prefect Server with the official Helm chart. - Create a Cloud Run work pool (hybrid). - Run a Prefect worker via the Helm worker chart (in Kubernetes) and set its worker type to
cloud-run
or
cloud-run-v2
. That worker will submit your flow runs as Cloud Run Jobs. - Alternatively: you can run the worker as a Cloud Run Service, but that’s deployed with gcloud, not Helm. Important notes - Push work pools (no worker) are Prefect Cloud only. When self-hosting, use a hybrid work pool with a running worker. - The worker does not need to run in Cloud Run — running it in your Kubernetes cluster via the Helm worker chart is a common/solid pattern. Links - Prefect Helm charts repo: prefect-helm - Server with Helm docs: Self-host with Helm - Worker chart (README + values): prefect-worker chart - Work pools: Work pools in Prefect 3.x - GCP worker guide: GCP worker guide Option A: Run the worker in Kubernetes via Helm (recommended for self-hosted) 1) Install Prefect Server - Add the Helm repo and install the server (per docs):
Copy code
helm repo add prefect <https://prefecthq.github.io/prefect-helm>
helm repo update

kubectl create namespace prefect
helm install prefect-server prefect/prefect-server -n prefect
- Access the UI (port-forward if needed) and confirm the API URL you’ll use (e.g., http://prefect-server.prefect.svc.cluster.local:4200/api or your ingress hostname). 2) Create a Cloud Run work pool (hybrid) - From a machine that can talk to your self-hosted API:
Copy code
export PREFECT_API_URL="http://<your-prefect-server>/api"
prefect work-pool create --type cloud-run-v2 my-cloud-run-pool
- In the UI, open the work pool and set the base job template (region, service account, CPU/memory, timeout, etc.). These defaults apply to all runs unless overridden. 3) Give the worker permissions in GCP - The service account used to submit Cloud Run Jobs needs: - roles/run.admin - roles/iam.serviceAccountUser on the “run-as” job service account - roles/artifactregistry.reader if your flow images are in Artifact Registry 4) Deploy the worker via the Helm worker chart - Create a values file (example for a self-hosted API + Cloud Run v2 worker type, using Workload Identity on GKE):
Copy code
worker:
  apiConfig: selfHostedServer
  selfHostedServerApiConfig:
    apiUrl: "<http://prefect-server.prefect.svc.cluster.local:4200/api>"
    basicAuth:
      enabled: false

  image:
    repository: prefecthq/prefect
    tag: "3-latest"
    pullPolicy: IfNotPresent

  config:
    # name is optional; if omitted the chart/worker will auto-generate one
    workPool: "my-cloud-run-pool"
    type: "cloud-run-v2"
    installPolicy: "always"   # auto-install integration for Cloud Run worker at startup

  # If using Workload Identity (recommended on GKE)
serviceAccount:
  create: true
  name: "prefect-worker"
  annotations:
    <http://iam.gke.io/gcp-service-account|iam.gke.io/gcp-service-account>: "prefect-worker@YOUR_GCP_PROJECT.iam.gserviceaccount.com"

# If not using Workload Identity, mount a GCP key instead:
# worker:
#   extraVolumes:
#     - name: gcp-credentials
#       secret:
#         secretName: gcp-credentials
#         defaultMode: 0400
#   extraVolumeMounts:
#     - name: gcp-credentials
#       mountPath: /var/secrets/google
#       readOnly: true
#   extraEnvVars:
#     - name: GOOGLE_APPLICATION_CREDENTIALS
#       value: /var/secrets/google/key.json
- Install the worker:
Copy code
helm install prefect-worker prefect/prefect-worker -n prefect -f worker-values.yaml
- Watch logs to confirm it connects and starts polling:
Copy code
kubectl logs -n prefect -l <http://app.kubernetes.io/name=prefect-worker|app.kubernetes.io/name=prefect-worker> -f
5) Deploy your flows to the Cloud Run work pool - Build/push your flow image to Artifact Registry and deploy with
prefect deploy
, or use
flow.deploy(...)
in code. Make sure the deployment targets
work_pool_name="my-cloud-run-pool"
. - The Cloud Run region, service account, etc. are best set in the work pool’s base job template (UI), but can also be provided as job_variables on the deployment if needed. Option B: Run the worker as a Cloud Run Service (not via Helm) - If you prefer the worker itself to run in Cloud Run:
Copy code
gcloud run deploy prefect-worker \
  --image=prefecthq/prefect:3-latest \
  --region us-central1 \
  --service-account <CALLER_SA> \
  --set-env-vars PREFECT_API_URL=http://<your-prefect-server>/api \
  --min-instances 1 \
  --args "prefect","worker","start","-p","my-cloud-run-pool","-t","cloud-run-v2","--install-policy","always","--with-healthcheck"
- This is useful if you want everything fully serverless; just note this is deployed with gcloud, not Helm. Common gotchas - Push work pools (no worker) are Cloud-only; self-hosted requires a hybrid work pool + worker. - Ensure the worker has GCP permissions to create/run Cloud Run Jobs and impersonate the run-as service account. - If using Artifact Registry, grant reader access to the job’s run-as service account for your repo. If you can share whether you want the worker to run inside your Kubernetes cluster (via Helm) or as a Cloud Run Service, I can tailor a minimal values.yaml or gcloud command for your exact setup.