<@ULVA73B9P> how to set workpool and block via con...
# marvin-ai
l
@Marvin how to set workpool and block via configuration file when deploying with helm
m
thought for 649.1 seconds
Great question. Short answer: - Work pool: set it in the prefect-worker Helm chart values. - Blocks: the chart does not create blocks; provision them separately (CLI/API/Terraform) or via a Kubernetes Job/Helm hook you add. Details and examples below. Where to set the work pool in values.yaml - The official prefect-worker chart expects a work pool name at
worker.config.workPool
and the worker type at
worker.config.type
(kubernetes, docker, etc.). Example values.yaml (Prefect Cloud)
Copy code
global:
  prefect:
    prefectApiUrl: <https://api.prefect.cloud/api/accounts/<ACCOUNT_ID>/workspaces/<WORKSPACE_ID>>

image:
  repository: prefecthq/prefect
  prefectTag: 3-latest
  pullPolicy: IfNotPresent

worker:
  replicaCount: 1
  apiConfig: cloud
  cloudApiConfig:
    apiKeySecret:
      name: prefect-api-key          # K8s secret you create with your API key
      key: key
  config:
    type: kubernetes
    workPool: my-k8s-pool
    # Optional: provide a base job template to set job namespace, SA, etc.
    # baseJobTemplate:
    #   configuration: |-
    #     { ... JSON from `prefect work-pool get-default-base-job-template --type kubernetes` ... }
Example values.yaml (self-hosted Server)
Copy code
image:
  repository: prefecthq/prefect
  prefectTag: 3-latest

worker:
  replicaCount: 1
  apiConfig: server
  serverApiConfig:
    apiUrl: <http://prefect-server.prefect.svc.cluster.local:4200/api>
    # uiUrl: <http://prefect-server.prefect.svc.cluster.local:4200>
  config:
    type: kubernetes
    workPool: my-k8s-pool
    # baseJobTemplate:
    #   configuration: |-
    #     { ...customized JSON... }
Notes on queues and tags - Prefect workers can be restricted to specific work queues via the worker start options. The Helm chart’s primary required value is the pool (
worker.config.workPool
); queue-level filtering is typically handled when starting the worker. - If you need to restrict queues from Helm, the pattern is to set the worker’s start flags accordingly. If your chart version does not expose explicit keys for queues/tags, you can: - Configure them on the pool/queues in Prefect UI/API, or - Override the container command/args in the chart (if supported by your version) to pass
--work-queue
flags. Customizing the Kubernetes job template (namespace, service account, etc.) - To fix job namespace, service account, and other run-time details for flow runs, pass a base job template to the worker. The chart supports embedding JSON or using
--set-file
. Example (inline JSON):
Copy code
worker:
  config:
    workPool: my-k8s-pool
    type: kubernetes
    baseJobTemplate:
      configuration: |-
        {
          "job_configuration": {
            "name": "{{ name }}",
            "namespace": "flows-namespace",
            "job_manifest": {
              "apiVersion": "batch/v1",
              "kind": "Job",
              "metadata": {
                "generateName": "{{ name }}-",
                "namespace": "flows-namespace"
              },
              "spec": {
                "template": {
                  "spec": {
                    "serviceAccountName": "flows-service-account",
                    "restartPolicy": "Never",
                    "containers": [
                      {
                        "name": "prefect-job",
                        "image": "{{ image }}",
                        "imagePullPolicy": "{{ image_pull_policy }}",
                        "args": "{{ command }}",
                        "env": "{{ env }}"
                      }
                    ]
                  }
                }
              }
            }
          }
        }
Can Helm create the work pool or blocks for me? - Work pools: Typically created/managed via CLI/UI/API/Terraform outside the chart. The worker chart requires
worker.config.workPool
. You can create/update pools in CI/CD with: -
prefect work-pool create "my-k8s-pool" --type kubernetes --base-job-template ./base-job-template.json
- Blocks: The chart does not create block types or block instances. Use one of: - A one-time Kubernetes Job (Helm post-install/upgrade hook) that runs Python/CLI to register block types and create block instances. - Terraform or your own bootstrap script. Example Helm post-install Job to provision blocks
Copy code
apiVersion: batch/v1
kind: Job
metadata:
  name: prefect-bootstrap-blocks
  annotations:
    "<http://helm.sh/hook|helm.sh/hook>": post-install,post-upgrade
    "<http://helm.sh/hook-delete-policy|helm.sh/hook-delete-policy>": before-hook-creation,hook-succeeded
spec:
  template:
    spec:
      serviceAccountName: prefect-worker  # or another SA with network access to Prefect
      restartPolicy: Never
      containers:
        - name: bootstrap
          image: prefecthq/prefect:3-latest
          env:
            - name: PREFECT_API_URL
              valueFrom:
                secretKeyRef:
                  name: prefect-api
                  key: PREFECT_API_URL
            - name: PREFECT_API_KEY
              valueFrom:
                secretKeyRef:
                  name: prefect-api
                  key: PREFECT_API_KEY
          command: ["/bin/sh","-c"]
          args:
            - |
              set -e
              # Register any block types you need (example: prefect-aws)
              prefect block register -m prefect_aws || true

              python - <<'PY'
              # Example: create/update an S3 block
              from prefect_aws.credentials import AwsCredentials
              from prefect_aws.s3 import S3Bucket

              creds = AwsCredentials.load("aws-creds") if "aws-creds" in [b.name for b in AwsCredentials.list()] else None
              if not creds:
                  creds = AwsCredentials(
                      aws_access_key_id="<<inject via secrets or IRSA>>",
                      aws_secret_access_key="<<inject via secrets or IRSA>>",
                      region_name="us-east-1",
                  )
              creds.save("aws-creds", overwrite=True)

              s3 = S3Bucket(bucket_name="my-bucket", credentials=creds)
              s3.save("my-s3", overwrite=True)
              PY
Links - Prefect Helm charts repo: prefect-helm on GitHub - Prefect worker chart docs and values: prefect-worker chart README, values.yaml - Kubernetes worker docs (includes Helm usage and base job template): Prefect docs: Deploy a worker using Helm - Work pools overview/how-to: Concepts: Work Pools, How to manage work pools A couple quick questions so I can tailor this precisely: - Which chart version are you using? - Are you on Prefect Cloud or self-hosted Server? - Which block types do you want to create (e.g., S3, GCS, Docker Registry, etc.)? I can give you an exact bootstrap Job snippet for those.