https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> My prefect server and a prefect worke...
# ask-marvin
r
@Marvin My prefect server and a prefect worker is running in k8s on google cloud, I would also like to have a worker outside k8s (on local infrastructure). How can I do this?
m
✅ thought for 447.5 seconds
Absolutely — you can run an additional worker on your local infrastructure alongside your Kubernetes-based Prefect Server and k8s worker. Assuming you’re on Prefect 3.x, here’s the quickest path: High-level - You’ll create a separate work pool for your non-k8s worker (pool types can’t be mixed). - Point the local machine at your Prefect Server’s API (running in GKE). - Start a Process (or Docker) worker on the local machine. - Deploy flows to that pool. Step-by-step 1) Point the local machine at your Prefect Server API - Ensure your local host can reach the Prefect API in GKE (typically exposed via LoadBalancer/Ingress on port 4200). - Set the API URL (note the /api suffix):
Copy code
export PREFECT_API_URL="http://<YOUR_GKE_SERVER_HOSTNAME_OR_IP>:4200/api"
prefect profile create local-worker
prefect profile use local-worker
prefect config set PREFECT_API_URL="$PREFECT_API_URL"
Docs: Settings &amp; profiles 2) Create a work pool for the local worker - Pools are single-type. Your Kubernetes pool cannot be shared by a Process worker.
Copy code
prefect work-pool create my-local-process-pool --type process
prefect work-pool ls
prefect work-pool inspect my-local-process-pool
Docs: Work pools and Workers 3) Start the local worker - Install Prefect and any flow dependencies in this machine’s Python environment first.
Copy code
pip install prefect
prefect worker start --pool my-local-process-pool --name my-local-worker
4) Deploy flows to the local pool Option A: Deploy via Python (recommended for 3.x)
Copy code
from prefect import flow

@flow(log_prints=True)
def my_flow():
    print("Hello from local worker!")

if __name__ == "__main__":
    my_flow.from_source(
        source="<https://github.com/your-org/your-repo.git>",     # or a local path
        entrypoint="flows/my_flow.py:my_flow",
    ).deploy(
        name="local-deployment",
        work_pool_name="my-local-process-pool",
    )
Docs: Deploy via Python Option B: Use 
prefect deploy
with a project
Copy code
prefect init --recipe process
# edit prefect.yaml to set work_pool.name: my-local-process-pool
prefect deploy
Docs: Define deployments with prefect.yaml Important notes - Network: Your local machine must be able to reach the GKE API endpoint (port 4200 by default). Verify with:
Copy code
curl http://<YOUR_GKE_SERVER_HOSTNAME_OR_IP>:4200/api/health
- Pool types can’t be mixed: keep k8s worker(s) on a Kubernetes pool and the local worker on a Process (or Docker) pool. - Code availability: For Process workers, make sure your code is either: - On the local machine (use a local path in 
.from_source(...)
), or - Pulled from Git, or - Pulled from blob storage Docs: Store/retrieve flow code - Dependencies: Ensure the local machine’s Python environment includes everything your flows need (pip install them in the same environment where the worker runs). Helpful docs - Work pools: Concepts - Workers: Concepts - Deploy via Python: Guide - Settings & profiles: Guide Questions to tailor the setup - Do you prefer a Process worker or a Docker worker on the local machine? - Is your Prefect Server exposed via LoadBalancer/Ingress, and can your local box reach it on port 4200? - How do you want your flow code delivered to the local machine (local path vs Git vs storage)?
r
@Marvin If I want to run a k8s worker on a k8s cluster running in docker desktop, how will that work?
m
✅ thought for 142.4 seconds
You’ve got two good ways to run a Kubernetes worker against a Docker Desktop Kubernetes cluster. Pick the one that fits your workflow. Option A — Easiest for local dev: run the worker as a local process using your docker-desktop kubeconfig - When the worker runs locally, it uses your kubeconfig to create Jobs in the docker-desktop cluster. - Steps: 1) Point Prefect at your API (Cloud or self-hosted):
Copy code
# Cloud
  export PREFECT_API_URL="<https://api.prefect.cloud/api/accounts/<ACCOUNT_ID>/workspaces/<WORKSPACE_ID>>"
  export PREFECT_API_KEY="<YOUR_KEY>"

  # Self-hosted on your machine
  export PREFECT_API_URL="<http://localhost:4200/api>"
2) Make sure kubectl is pointed at docker-desktop:
Copy code
kubectl config use-context docker-desktop
  kubectl get nodes
3) Create a Kubernetes work pool (pool types are per-infra):
Copy code
prefect work-pool create my-dd-k8s-pool --type kubernetes --set namespace=default --set image_pull_policy=IfNotPresent
4) Start the worker locally:
Copy code
prefect worker start --pool my-dd-k8s-pool --type kubernetes
5) Deploy flows to that pool (via 
.from_source(...).deploy(...)
or 
prefect deploy
), then run them. - Notes for Docker Desktop: - Use 
image_pull_policy=IfNotPresent
so local images aren’t pulled every time. - If your Prefect Server runs on the host, Jobs can’t reach 
<http://localhost:4200/api>
from inside the pod; use 
<http://host.docker.internal:4200/api>
. Prefect’s k8s integration will rewrite localhost to 
host.docker.internal
automatically in many cases, but it’s safest to set it explicitly in the worker/pool config. - If you use private images, create an imagePullSecret and reference it in the pool’s base job template. - RBAC: Docker Desktop is permissive by default, but if you use a non-default namespace/service account, give it permissions to create/list/watch/delete Jobs and read pods/logs. Option B — Run the worker inside the cluster (Helm) - This is closer to production and removes kubeconfig dependence. You deploy a “Prefect worker” pod to your docker-desktop cluster that creates Jobs for your flow runs. - Steps (high level): 1) Create a Kubernetes work pool 
--type kubernetes
(as above). 2) Add Helm repo and prepare values with your API config and pool name. 3) Install:
Copy code
helm repo add prefect <https://prefecthq.github.io/prefect-helm>
  helm repo update
  helm install prefect-worker prefect/prefect-worker -n prefect --create-namespace -f values.yaml
- In your values: - For Prefect Cloud: set accountId, workspaceId, api key secret, and workPool. - For self-hosted: set 
selfHostedServerApiConfig.apiUrl: <http://host.docker.internal:4200/api>
if Server is on your host. - Keep 
image_pull_policy: IfNotPresent
for local images. Common pitfalls and tips - API reachability from pods: use 
host.docker.internal
to reach host services from inside Docker Desktop pods. - Images: If you build custom flow images locally, tag them and set that image in the work pool or deployment; 
IfNotPresent
helps reuse local cache. - Resources: Docker Desktop memory defaults are small—bump memory/CPU in Docker Desktop settings if Jobs are evicted or stuck scheduling. - Storage: If Jobs need persistent storage, add volumes to your pool’s base job template (e.g., emptyDir or hostPath for simple local dev). - RBAC (if needed): bind a Role allowing jobs create/get/list/watch/update/patch/delete and pods get/list/watch/logs in your target namespace. Docs for deeper reference - Work pools: Work pools - Workers: Workers - Kubernetes how-to: Run on Kubernetes - Deploy via Python: Deploy via Python - Settings & profiles: Settings &amp; profiles
If you share whether you’re targeting Prefect Cloud or a self-hosted Server (and where it’s running), I can provide a ready-to-run values.yaml or the exact pool job template you’ll need.
r
@Marvin If I make a docker worker, how can i set gcp credentials to run with a service account? Can i use the prefect gcp credentials block?
m
✅ thought for 276.2 seconds
Short answer: yes—you can run Docker flow runs with a GCP service account and you can use the Prefect GCP Credentials block. There are two common patterns depending on how your code authenticates. Option A (recommended): Use the GcpCredentials block in your flow code - Store the service account JSON in a GcpCredentials block (encrypted in Prefect). - Load the block in your flow and pass it to prefect-gcp tasks or use it to create authenticated clients. Example: create and use the block
Copy code
# One-time setup (run anywhere with Prefect auth)
from prefect_gcp import GcpCredentials

gcp_credentials = GcpCredentials(
    service_account_info={...}  # your service account JSON dict
)
gcp_credentials.save("my-gcp-creds")  # save as a Prefect block
Use it in a flow
Copy code
from prefect import flow
from prefect_gcp import GcpCredentials
from prefect_gcp.bigquery import bigquery_query

@flow
def my_flow():
    gcp_creds = GcpCredentials.load("my-gcp-creds")
    result = bigquery_query(
        query="SELECT 1",
        gcp_credentials=gcp_creds,
        to_dataframe=False,
    )
    return result
Notes: - This pattern does not require setting GOOGLE_APPLICATION_CREDENTIALS. - Make sure your image includes the GCP extras: install 
prefect[gcp]
and any google-cloud libs you use. Option B: If your code relies on GOOGLE_APPLICATION_CREDENTIALS (ADC) If you’re using google-cloud libraries directly and expect Application Default Credentials, set the env var inside the Docker job. You can do that in a few ways: 1) Mount a key file into the container and point GOOGLE_APPLICATION_CREDENTIALS at it - In your deployment’s work_pool job_variables:
Copy code
work_pool:
  name: my-docker-pool
  job_variables:
    env:
      GOOGLE_APPLICATION_CREDENTIALS: /opt/creds/sa.json
    volumes:
      - "/host/path/sa.json:/opt/creds/sa.json:ro"
2) Store the JSON as a Prefect Secret and write the file at runtime in your flow
Copy code
from prefect import flow
from prefect.blocks.system import Secret
import os, tempfile

@flow
def my_flow():
    sa_json = Secret.load("gcp-sa-json").get()  # Secret holds the JSON
    with tempfile.NamedTemporaryFile(delete=False, suffix=".json") as f:
        f.write(sa_json.encode())
        os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = f.name

    # Now google-cloud clients will pick up ADC
    from google.cloud import storage
    storage.Client().list_buckets()
3) Pass the JSON into the container via an env var and write to a file at runtime - Put a base64 or plain JSON value into 
job_variables.env
and write it to 
/tmp/sa.json
in your flow before initializing clients, then set 
GOOGLE_APPLICATION_CREDENTIALS
to that path. How this fits with Docker workers - You’ll create a Docker work pool and start a Docker worker:
Copy code
prefect work-pool create --type docker my-docker-pool
prefect worker start --pool my-docker-pool
- For Option A, nothing special is required in job env—your flow loads the GcpCredentials block and authenticates. - For Option B, set env/volumes in the deployment’s 
work_pool.job_variables
so the container has the credentials file or JSON available. Quick guidance - Prefer the GcpCredentials block when you’re using prefect-gcp tasks or want to construct clients from the block—it’s portable and avoids managing files in containers. - Use GOOGLE_APPLICATION_CREDENTIALS only if your code or libraries specifically expect ADC. Would you like an example aligned to your exact setup? Are you: - using prefect-gcp tasks (BigQuery, GCS, etc.), or - using google-cloud libraries directly that expect ADC?
r
@Marvin - How can I configure my workpool to use my docker registry credentials block?
m
✅ thought for 244.5 seconds
Yes. For a Docker work pool, you can set your DockerRegistryCredentials block as the default “registry credentials” so all flow runs can pull private images. Two easy ways: Option 1 — Set default job variables in the UI - Create/save your block (e.g., 
DockerRegistryCredentials
named 
my-registry
). - In the Prefect UI: Work Pools -> your Docker pool -> Edit -> Default job variables -> set “Registry credentials” to your block 
my-registry
-> Save. Option 2 — Update the work pool’s base job template via CLI 1) Export the default Docker template
Copy code
prefect work-pool get-default-base-job-template --type docker --file docker.json
2) Edit docker.json and set the registry credentials to your block reference by replacing this line in `job_configuration`:
Copy code
"registry_credentials": "{{ registry_credentials }}"
with:
Copy code
"registry_credentials": "{{ prefect.blocks.docker-registry-credentials.my-registry }}"
3) Update the work pool
Copy code
prefect work-pool update my-docker-pool --base-job-template docker.json
Per-deployment alternative - You can also set it per deployment in `job_variables`:
Copy code
my_flow.deploy(
  name="private-image",
  work_pool_name="my-docker-pool",
  image="<http://myregistry.example.com/my-image:latest|myregistry.example.com/my-image:latest>",
  job_variables={
    "registry_credentials": "{{ prefect.blocks.docker-registry-credentials.my-registry }}"
  },
)
Notes - The block should contain 
username
, 
password
(token), and 
registry_url
(e.g., 
<http://index.docker.io|index.docker.io>
, 
<http://myregistry.azurecr.io|myregistry.azurecr.io>
, or your ECR URL). - If you prefer, you can skip blocks entirely by running 
docker login
on the worker host; the worker will use the host’s Docker config. Blocks are more portable and easier to manage per pool/deployment.
r
@Marvin - The option to select a docker registry block for my docker work pool is not showing
m
✅ thought for 208.4 seconds
Got it — if the “Registry credentials” selector isn’t showing for your Docker work pool, it’s usually one of these: 1) The Docker block type isn’t registered in your workspace yet - Install the Docker extra and register the block type against your workspace:
Copy code
# ensure the docker integration is available
uv run --with 'prefect[docker]' prefect block register -m prefect_docker.credentials
- Verify it’s available:
Copy code
prefect block type ls | grep docker-registry-credentials
- Then create a block instance (UI: Blocks -> + New -> Docker Registry Credentials) or via CLI:
Copy code
prefect block create docker-registry-credentials
2) Your work pool’s base job template doesn’t include the registry_credentials field - Export the default Docker template and check that it includes both the job_configuration entry and the variable:
Copy code
prefect work-pool get-default-base-job-template --type docker --file docker.json
# Look for:
# "registry_credentials": "{{ registry_credentials }}" in job_configuration
# and a variables definition for DockerRegistryCredentials
- If your pool was created before this was added or uses a custom template, update it:
Copy code
prefect work-pool update <your-pool> --base-job-template docker.json
3) You’re looking in the wrong place - The selector appears: - In Work Pool -> Edit -> Default job variables (for pool-wide defaults), or - In a Deployment’s job variables (per deployment). - It will only render if: - The block type is registered, and - The work pool’s base job template has the 
registry_credentials
field. Quick checks - Confirm Prefect 3.x and upgrade to latest to ensure your Docker template includes 
registry_credentials
. - Confirm you’re authenticated to the correct workspace when running 
prefect block register
. Workaround if the selector still doesn’t show - You can still set it explicitly in job variables using a block reference string:
Copy code
"registry_credentials": "{{ prefect.blocks.docker-registry-credentials.my-registry }}"
If you share: - Prefect version, - Whether you’re on Cloud or self-hosted, - Your pool name and whether it uses a custom base job template, I can give exact commands to get the selector to show up.
r
@Marvin My template looks like this: (algo-pipeline) PS C:\repos\algo-pipeline> uv run prefect work-pool get-default-base-job-template --type docker { "job_configuration": { "command": "{{ command }}", "env": "{{ env }}", "labels": "{{ labels }}", "name": "{{ name }}", "image": "{{ image }}", "registry_credentials": "{{ registry_credentials }}", "image_pull_policy": "{{ image_pull_policy }}", "networks": "{{ networks }}", "network_mode": "{{ network_mode }}", "auto_remove": "{{ auto_remove }}", "volumes": "{{ volumes }}", "stream_output": "{{ stream_output }}", "mem_limit": "{{ mem_limit }}", "memswap_limit": "{{ memswap_limit }}", "privileged": "{{ privileged }}", "container_create_kwargs": "{{ container_create_kwargs }}" }, "variables": { "definitions": { "DockerRegistryCredentials": { "additionalProperties": true, "block_schema_references": {}, "block_type_slug": "docker-registry-credentials", "description": "Store credentials for interacting with a Docker Registry.", "properties": { "username": { "description": "The username to log into the registry with.", "title": "Username", "type": "string" }, "password": { "description": "The password to log into the registry with.", "format": "password", "title": "Password", "type": "string", "writeOnly": true }, "registry_url": { "description": "The URL to the registry. Generally, \"http\" or \"https\" can be omitted.", "examples": [ "index.docker.io" ], "title": "Registry Url", "type": "string" }, "reauth": { "default": true, "description": "Whether or not to reauthenticate on each interaction.", "title": "Reauth", "type": "boolean" } }, "required": [ "username", "password", "registry_url" ], "secret_fields": [ "password" ], "title": "DockerRegistryCredentials", "type": "object" } }, "description": "Configuration class used by the Docker worker.\n\nAn instance of this class is passed to the Docker worker's 
run
method\nfor each flow run. It contains all the information necessary to execute the\nflow run as a Docker container.\n\nAttributes:\n name: The name to give to created Docker containers.\n command: The command executed in created Docker containers to kick off\n flow run execution.\n env: The environment variables to set in created Docker containers.\n labels: The labels to set on created Docker containers.\n image: The image reference of a container image to use for created jobs.\n If not set, the latest Prefect image will be used.\n image_pull_policy: The image pull policy to use when pulling images.\n networks: Docker networks that created containers should be connected to.\n network_mode: The network mode for the created containers (e.g. host, bridge).\n If 'networks' is set, this cannot be set.\n auto_remove: If set, containers will be deleted on completion.\n volumes: Docker volumes that should be mounted in created containers.\n stream_output: If set, the output from created containers will be streamed\n to local standard output.\n mem_limit: Memory limit of created containers. Accepts a value\n with a unit identifier (e.g. 100000b, 1000k, 128m, 1g.) If a value is\n given without a unit, bytes are assumed.\n memswap_limit: Total memory (memory + swap), -1 to disable swap. Should only be\n set if 
mem_limit
is also set. If 
mem_limit
is set, this defaults to\n allowing the container to use as much swap as memory. For example, if\n 
mem_limit
is 300m and 
memswap_limit
is not set, containers can use\n 600m in total of memory and swap.\n privileged: Give extended privileges to created containers.\n container_create_kwargs: Extra args for docker py when creating container.", "properties": { "command": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The command to use when starting a flow run. In most cases, this should be left blank and the command will be automatically generated by the worker.", "title": "Command" }, "env": { "additionalProperties": { "anyOf": [ { "type": "string" }, { "type": "null" } ] }, "description": "Environment variables to set when starting a flow run.", "title": "Environment Variables", "type": "object" }, "labels": { "additionalProperties": { "type": "string" }, "description": "Labels applied to infrastructure created by the worker using this job configuration.", "title": "Labels", "type": "object" }, "name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "Name given to infrastructure created by the worker using this job configuration.", "title": "Name" }, "image": { "description": "The image reference of a container image to use for created jobs. If not set, the latest Prefect image will be used.", "examples": [ "docker.io/prefecthq/prefect:3-latest" ], "title": "Image", "type": "string" }, "registry_credentials": { "anyOf": [ { "$ref": "#/definitions/DockerRegistryCredentials" }, { "type": "null" } ], "default": null, "description": "Credentials for logging into a Docker registry to pull images from." }, "image_pull_policy": { "anyOf": [ { "enum": [ "IfNotPresent", "Always", "Never" ], "type": "string" }, { "type": "null" } ], "default": null, "description": "The image pull policy to use when pulling images.", "title": "Image Pull Policy" }, "networks": { "description": "Docker networks that created containers should be connected to.", "items": { "type": "string" }, "title": "Networks", "type": "array" }, "network_mode": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The network mode for the created containers (e.g. host, bridge). If 'networks' is set, this cannot be set.", "title": "Network Mode" }, "auto_remove": { "default": false, "description": "If set, containers will be deleted on completion.", "title": "Auto Remove", "type": "boolean" }, "volumes": { "description": "A list of volume to mount into created containers.", "examples": [ "/my/local/path:/path/in/container" ], "items": { "type": "string" }, "title": "Volumes", "type": "array" }, "stream_output": { "default": true, "description": "If set, the output from created containers will be streamed to local standard output.", "title": "Stream Output", "type": "boolean" }, "mem_limit": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "Memory limit of created containers. Accepts a value with a unit identifier (e.g. 100000b, 1000k, 128m, 1g.) If a value is given without a unit, bytes are assumed.", "title": "Memory Limit" }, "memswap_limit": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "Total memory (memory + swap), -1 to disable swap. Should only be set if 
mem_limit
is also set. If 
mem_limit
is set, this defaults toallowing the container to use as much swap as memory. For example, if 
mem_limit
is 300m and 
memswap_limit
is not set, containers can use 600m in total of memory and swap.", "title": "Memory Swap Limit" }, "privileged": { "default": false, "description": "Give extended privileges to created container.", "title": "Privileged", "type": "boolean" }, "container_create_kwargs": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "default": null, "description": "Configuration for containers created by workers. See the [
docker-py
documentation](https://docker-py.readthedocs.io/en/stable/containers.html) for accepted values.", "title": "Container Configuration" } }, "type": "object" } }
m
Your message was too long, here's your message at the allowed limit: My template looks like this (algo-pipeline) PS C \repos\algo-pipeline> uv run prefect work-pool get-default-base-job-template --type docker { job_configuration { command {{ command }} env {{ env }} labels {{ labels }} name {{ name }} image {{ image }} registry_credentials {{ registry_credentials }} image_pull_policy {{ image_pull_policy }} networks {{ networks }} network_mode {{ network_mode }} auto_remove {{ auto_remove }} volumes {{ volumes }} stream_output {{ stream_output }} mem_limit {{ mem_limit }} memswap_limit {{ memswap_limit }} privileged {{ privileged }} container_create_kwargs {{ container_create_kwargs }} } variables { definitions { DockerRegistryCredentials { additionalProperties true block_schema_references {} block_type_slug docker-registry-credentials description Store credentials for interacting with a Docker Registry properties { username { description The username to log into the registry with title Username type string } password { description The password to log into the registry with format password title Password type string writeOnly true } registry_url { description The URL to the registry Generally \ http\ or \ https\ can be omitted examples [ <http //index docker io|index docker io> ] title Registry Url type string } reauth { default true description Whether or not to reauthenticate on each interaction title Reauth type boolean } } required [ username password registry_url ] secret_fields [ password ] title DockerRegistryCredentials type object } } description Configuration class used by the Docker worker \n\nAn instance of this class is passed to the Docker worker's 
run
method\nfor each flow run It contains all the information necessary to execute the\nflow run as a Docker container \n\nAttributes \n name The name to give to created Docker containers \n command The command executed in created Docker containers to kick off\n flow run execution \n env The environment variables to set in created Docker containers \n labels The labels to set on created Docker containers \n image The image reference of a container image to use for created jobs \n If not set the latest Prefect image will be used \n image_pull_policy The image pull policy to use when pulling images \n networks Docker networks that created containers should be connected to \n network_mode The network mode for the created containers (e g host bridge) \n If 'networks' is set this cannot be set \n auto_remove If set containers will be deleted on completion \n volumes Docker volumes that should be mounted in created containers \n stream_output If set the output from created containers will be streamed\n to local standard output \n mem_limit Memory limit of created containers Accepts a value\n with a unit identifier (e g 100000b 1000k 128m 1g ) If a value is\n given without a unit bytes are assumed \n memswap_limit Total memory (memory + swap) -1 to disable swap Should only be\n set if 
mem_limit
is also set If 
mem_limit
is set this defaults to\n allowing the container to use as much swap as memory For example if\n 
mem_limit
is 300m and 
memswap_limit
is not set containers can use\n 600m in total of memory and swap \n privileged Give extended privileges to created containers \n
r
@Marvin my docker related config in my base job template looks like this: "variables": { "definitions": { "DockerRegistryCredentials": { "additionalProperties": true, "block_schema_references": {}, "block_type_slug": "docker-registry-credentials", "description": "Store credentials for interacting with a Docker Registry.", "properties": { "username": { "description": "The username to log into the registry with.", "title": "Username", "type": "string" }, "password": { "description": "The password to log into the registry with.", "format": "password", "title": "Password", "type": "string", "writeOnly": true }, "registry_url": { "description": "The URL to the registry. Generally, \"http\" or \"https\" can be omitted.", "examples": [ "index.docker.io" ], "title": "Registry Url", "type": "string" }, "reauth": { "default": true, "description": "Whether or not to reauthenticate on each interaction.", "title": "Reauth", "type": "boolean" } }, "required": [ "username", "password", "registry_url" ], "secret_fields": [ "password" ], "title": "DockerRegistryCredentials", "type": "object" } }, And "registry_credentials": { "anyOf": [ { "$ref": "#/definitions/DockerRegistryCredentials" }, { "type": "null" }
m
✅ thought for 43.9 seconds
Thanks for sharing the template snippets — those look correct for defining a DockerRegistryCredentials variable. If the “Registry credentials” block picker still isn’t showing in the UI, it’s usually one of these: 1) The block type isn’t registered in your workspace - Register the Docker block type (ensure the docker extra is available):
Copy code
uv run --with 'prefect[docker]' prefect block register -m prefect_docker.credentials
- Verify it’s registered:
Copy code
prefect block type ls | grep docker-registry-credentials
2) There isn’t a block instance yet - The UI only shows the “Use a block” picker if at least one block of that type exists. - Create one:
Copy code
prefect block create docker-registry-credentials
- Or create via UI (Blocks -> + New -> Docker Registry Credentials), then refresh the work pool editor. 3) The base job template is missing the job_configuration reference - Make sure the job_configuration includes the field:
Copy code
"job_configuration": {
  ...
  "registry_credentials": "{{ registry_credentials }}",
  ...
}
- If it’s missing, add it and update the pool:
Copy code
prefect work-pool update <your-pool> --base-job-template docker.json
4) You’re looking in the wrong place in the UI - The selector appears either: - In Work Pools -> your Docker pool -> Edit -> Default job variables, or - In a deployment’s Job Variables. - It won’t show if the block type isn’t registered or there are no block instances. Quick diagnostics - Do you see the block type?
Copy code
prefect block type ls | grep docker-registry-credentials
- Do you have a block instance?
Copy code
prefect block ls | grep docker-registry-credentials
Workarounds - You can still set it directly as a block reference string in job variables (even if the UI picker isn’t visible):
Copy code
"registry_credentials": "{{ prefect.blocks.docker-registry-credentials.my-registry }}"
- Or inline credentials (less portable):
Copy code
"registry_credentials": {
  "username": "myuser",
  "password": "mypat_or_token",
  "registry_url": "<http://myregistry.example.com|myregistry.example.com>",
  "reauth": true
}
If you can share: - Prefect version (and whether Cloud or self-hosted), - Whether 
docker-registry-credentials
shows in 
prefect block type ls
, - Whether a block instance exists, I’ll tailor the exact steps to your setup.
r
@Marvin - What is the username and password I should supply?
m
✅ thought for 52.9 seconds
Short version: use whatever your registry expects for docker login. The block’s fields map 1:1 to 
docker login -u <username> -p <password> <registry>
. Common registries and what to put: - Docker Hub - username: your Docker Hub username - password: a Docker Hub Personal Access Token (recommended), not your account password - registry_url: index.docker.io - GitHub Container Registry (GHCR) - username: your GitHub username - password: a GitHub PAT with read:packages (and org permissions if needed) - registry_url: ghcr.io - Google Container/Artifact Registry - Option 1 (service account JSON, long‑lived): - username: _json_key - password: the full JSON key contents - registry_url: gcr.io, us.gcr.io, eu.gcr.io, asia.gcr.io, or LOCATION-docker.pkg.dev (e.g., us-central1-docker.pkg.dev) - Option 2 (short‑lived token): - username: oauth2accesstoken - password: output of 
gcloud auth print-access-token
- registry_url: as above - Note: token rotates frequently—better for host-level docker login than a block. - Azure Container Registry (ACR) - If Admin user enabled: - username: the registry’s admin username (often the registry name) - password: the admin password - If using a Service Principal: - username: the App Registration’s client ID (appId) - password: the client secret - registry_url: myregistry.azurecr.io - AWS Elastic Container Registry (ECR) - username: AWS - password: the value from 
aws ecr get-login-password --region <region>
(expires ~12 hours) - registry_url: <account>.dkr.ecr.<region>.amazonaws.com - Important: Because the password is short‑lived, prefer doing 
docker login
on the worker host or using the AWS ECR credential helper instead of storing in a block. - GitLab Container Registry - With a Deploy Token: - username: gitlab+deploy-token-<id> - password: the deploy token value - With a PAT: - username: your GitLab username - password: a PAT with read_registry - registry_url: registry.gitlab.com (or your self-hosted domain) Tips - Provide the registry host without scheme (e.g., index.docker.io, ghcr.io, myregistry.azurecr.io). - Use read‑only/PAT/robot credentials when possible. - For registries with short‑lived tokens (ECR, gcloud access tokens), host‑level 
docker login
or credential helpers are more reliable than storing in a block. Tell me which registry you’re using and I’ll give you the exact values and a ready-to-paste block creation snippet.
r
@Marvin I want to pull from Google Artifact Registry. Should my config then be: "registry_credentials": { "username": "_json_key", "password": gcp-sa-key.json "registry_url": "europe-west3-docker.pkg.dev", "reauth": true }
m
✅ thought for 30.5 seconds
Almost — two fixes: - password must be the full JSON key contents, not a file path - registry_url should be just the hostname (no scheme, no Slack link markup) For Google Artifact Registry with a service account JSON key, use:
Copy code
"registry_credentials": {
  "username": "_json_key",
  "password": "{...the entire service account JSON key...}",
  "registry_url": "europe-west3-docker.pkg.dev",
  "reauth": true
}
Notes - Don’t pass 
gcp-sa-key.json
as a path; paste the JSON content itself into the password field. The block stores it encrypted. - Make sure your image name matches GAR’s format: - 
europe-west3-docker.pkg.dev/PROJECT_ID/REPOSITORY/IMAGE:TAG
- The service account needs at least Artifact Registry Reader on the repo or project. Alternative (short‑lived token) - username: 
oauth2accesstoken
- password: output of 
gcloud auth print-access-token
- registry_url: 
europe-west3-docker.pkg.dev
- This expires quickly, so it’s better for host-level 
docker login
than storing in a block. If you’d like, I can generate a ready-to-save DockerRegistryCredentials block for your project/repo—just share your GAR host and whether you’ll paste the JSON key or prefer using an access token.
r
@Marvin Can you create a base job template for a docker worker that should accept environment variables and mounts?
m
✅ thought for 50.0 seconds
Absolutely — here’s a ready-to-use Docker work pool base job template that accepts environment variables and volume mounts. It also supports private registry credentials and other common options. Save this as docker.json: ``` { "job_configuration": { "command": "{{ command }}", "name": "{{ name }}", "labels": "{{ labels }}", "image": "{{ image }}", "registry_credentials": "{{ registry_credentials }}", "image_pull_policy": "{{ image_pull_policy }}", "env": "{{ env }}", "volumes": "{{ volumes }}", "networks": "{{ networks }}", "network_mode": "{{ network_mode }}", "auto_remove": "{{ auto_remove }}", "stream_output": "{{ stream_output }}", "mem_limit": "{{ mem_limit }}", "memswap_limit": "{{ memswap_limit }}", "privileged": "{{ privileged }}", "container_create_kwargs": "{{ container_create_kwargs }}" }, "variables": { "definitions": { "DockerRegistryCredentials": { "additionalProperties": true, "block_schema_references": {}, "block_type_slug": "docker-registry-credentials", "description": "Store credentials for interacting with a Docker Registry.", "properties": { "username": { "description": "The username to log into the registry with.", "title": "Username", "type": "string" }, "password": { "description": "The password to log into the registry with.", "format": "password", "title": "Password", "type": "string", "writeOnly": true }, "registry_url": { "description": "The URL/host for the registry (no scheme).", "examples": ["index.docker.io", "ghcr.io", "europe-west3-docker.pkg.dev"], "title": "Registry Url", "type": "string" }, "reauth": { "default": true, "description": "Whether or not to reauthenticate on each interaction.", "title": "Reauth", "type": "boolean" } }, "required": ["username", "password", "registry_url"], "secret_fields": ["password"], "title": "DockerRegistryCredentials", "type": "object" } }, "type": "object", "properties": { "image": { "type": "string", "title": "Image", "description": "Full image reference, e.g. registry/repo/image:tag" }, "image_pull_policy": { "type": "string", "title": "Image Pull Policy", "enum": ["IfNotPresent", "Always", "Never"], "default": "IfNotPresent" }, "env": { "type": "object", "title": "Environment Variables", "description": "Key-value env vars to set in the container.", "additionalProperties": { "type": ["string", "null"] }, "default": {} }, "volumes": { "type": "array", "title": "Volumes", "description": "Docker volume bind strings, e.g. '/host/containerro'.", "items": { "type": "string" }, "default": [] }, "networks": { "type": "array", "title": "Networks", "items": { "type": "string" }, "default": [] }, "network_mode": { "type": ["string", "null"], "title": "Network Mode", "default": null }, "auto_remove": { "type": "boolean", "title": "Auto Remove", "default": true }, "stream_output": { "type": "boolean", "title": "Stream Output", "default": true }, "mem_limit": { "type": ["string", "null"], "title": "Memory Limit", "description": "e.g. 512m, 1g; bytes if unit omitted.", "default": null }, "memswap_limit": { "type": ["string", "null"], "title": "Memory Swap Limit", "default": null },
"privileged": { "type": "boolean", "title": "Privileged", "default": false }, "labels": { "type": "object", "title": "Labels", "additionalProperties": { "type": "string" }, "default": {} }, "registry_credentials": { "title": "Registry credentials", "anyOf": [ { "$ref": "#/definitions/DockerRegistryCredentials" }, { "type": "null" } ], "default": null }, "container_create_kwargs": { "type": ["object", "null"], "title": "Container Configuration", "description": "Extra kwargs for docker.containers.create().", "default": null }, "name": { "type": "string", "title": "Container Name", "default": "prefect-job" } }, "required": ["image"] } }
Copy code
How to apply it
- Update your existing Docker work pool:
prefect work-pool update <your-docker-pool> --base-job-template docker.json
Copy code
How to use env vars and mounts in a deployment or pool defaults
- Pool defaults (UI): Work Pools -> your pool -> Edit -> Default job variables:
env: GCP_PROJECT: my-project FOO: bar volumes: - "/host/path/container/pathro" - "named_volume:/data"
Copy code
- Per-deployment job variables:
my_flow.deploy( name="with-env-and-mounts", work_pool_name="<your-docker-pool>", image="europe-west3-docker.pkg.dev/PROJECT/REPO/IMAGE:TAG", job_variables={ "env": { "ENV_ONE": "1", "GOOGLE_APPLICATION_CREDENTIALS": "/secrets/sa.json" }, "volumes": [ "/secrets/on/host/sa.json/secrets/sa.jsonro" ] } ) ``` If you want me to include specific defaults (e.g., a particular mount path or common envs), tell me and I’ll pre-populate them in the template.
r
@Marvin How can I use the Docker Registry Credentials block and set it for a docker worker?
m
✅ thought for 30.1 seconds
Here’s a concise, end‑to‑end way to use a Docker Registry Credentials block with a Docker work pool (Prefect 3.x): 1) Make sure the Docker block type is registered - Install the Docker extra and register the block type so it appears in your workspace:
Copy code
uv run --with 'prefect[docker]' prefect block register -m prefect_docker.credentials
- Verify it’s there:
Copy code
prefect block type ls | grep docker-registry-credentials
2) Create a Docker Registry Credentials block - UI: Blocks -> + New -> Docker Registry Credentials - Or CLI:
Copy code
prefect block create docker-registry-credentials
- Or Python:
Copy code
from prefect_docker.credentials import DockerRegistryCredentials
import asyncio

async def main():
    creds = DockerRegistryCredentials(
        username="...",
        password="...",            # token/secret
        registry_url="..."         # e.g. <http://index.docker.io|index.docker.io>, <http://ghcr.io|ghcr.io>, europe-west3-docker.pkg.dev
    )
    await creds.save("my-registry", overwrite=True)

asyncio.run(main())
3) Set the block for your Docker work pool (choose one) - Option A: UI (pool-wide default) - Work Pools -> your Docker pool -> Edit -> Default job variables -> set “Registry credentials” to your block (e.g., my-registry) -> Save. - Option B: CLI via base job template 1. Export the default template:
Copy code
prefect work-pool get-default-base-job-template --type docker --file docker.json
2. In docker.json, set:
Copy code
"job_configuration": {
    ...
    "registry_credentials": "{{ prefect.blocks.docker-registry-credentials.my-registry }}",
    ...
  }
3. Apply it:
Copy code
prefect work-pool update <your-docker-pool> --base-job-template docker.json
- Option C: Per-deployment job variables - In prefect.yaml:
Copy code
deployments:
    - name: private-image
      entrypoint: flows/my_flow.py:my_flow
      work_pool:
        name: <your-docker-pool>
        job_variables:
          image: "europe-west3-docker.pkg.dev/PROJECT/REPO/IMAGE:TAG"
          registry_credentials: "{{ prefect.blocks.docker-registry-credentials.my-registry }}"
- Or in Python:
Copy code
my_flow.deploy(
    name="private-image",
    work_pool_name="<your-docker-pool>",
    image="europe-west3-docker.pkg.dev/PROJECT/REPO/IMAGE:TAG",
    job_variables={
      "registry_credentials": "{{ prefect.blocks.docker-registry-credentials.my-registry }}"
    },
  )
How it works at runtime - The Docker worker receives 
registry_credentials
, logs into the registry, then pulls your image before starting the container. Notes - For Google Artifact Registry, use: - username: 
_json_key
- password: the full JSON key contents (not a file path) - registry_url: e.g., 
europe-west3-docker.pkg.dev
- If you prefer not to use blocks, you can 
docker login
on the worker host and the worker will use the host’s Docker config, but blocks are more portable and work across multiple workers. If you share your registry and pool name, I can give you the exact JSON snippet or commands tailored to your setup.
r
@Marvin If I am to use the Docker Registry Credentials block where my artifact registry is in GCP, how should i set the credentials=
m
✅ thought for 26.0 seconds
For Google Artifact Registry (GAR), set the Docker Registry Credentials block to use a service account JSON key: - username: 
_json_key
- password: the entire service account JSON key contents (as a single string) - registry_url: the GAR host for your region, e.g. 
europe-west3-docker.pkg.dev
(no scheme) Example: create/save the block
Copy code
from prefect_docker.credentials import DockerRegistryCredentials
import asyncio, json

async def main():
    # If you have the JSON as a dict
    sa_dict = {...}  # your service account key dict
    creds = DockerRegistryCredentials(
        username="_json_key",
        password=json.dumps(sa_dict),  # full JSON as a string
        registry_url="europe-west3-docker.pkg.dev",
    )
    await creds.save("gar-registry", overwrite=True)

asyncio.run(main())
Use it with a Docker work pool (so the worker can pull your image) - Pool default (UI): Work Pools -> your Docker pool -> Edit -> Default job variables -> set Registry credentials to 
gar-registry
. - Or in a deployment:
Copy code
my_flow.deploy(
  name="uses-gar-image",
  work_pool_name="my-docker-pool",
  image="europe-west3-docker.pkg.dev/PROJECT_ID/REPOSITORY/IMAGE:TAG",
  job_variables={
    "registry_credentials": "{{ prefect.blocks.docker-registry-credentials.gar-registry }}"
  },
)
Use it in the push step (so your CI can push your image)
Copy code
push:
  - prefect_docker.deployments.steps.push_docker_image:
      requires: prefect-docker
      image_name: "europe-west3-docker.pkg.dev/PROJECT_ID/REPOSITORY/IMAGE"
      tag: "latest"
      credentials: "{{ prefect.blocks.docker-registry-credentials.gar-registry }}"
Notes - Ensure the service account has Artifact Registry roles: - Pulling: roles/artifactregistry.reader (repo or project scope) - Pushing: roles/artifactregistry.writer - Image references must use the GAR host and path: - 
LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/IMAGE:TAG
- Alternative (short‑lived): username 
oauth2accesstoken
, password from 
gcloud auth print-access-token
, registry_url 
LOCATION-docker.pkg.dev
. Because tokens expire quickly, this is better for host-level 
docker login
than storing in a block.
r
@Marvin For task caching and a docker worker, how do i set the cache location?
m
✅ thought for 203.6 seconds
Great question. In Prefect 3.x, “task caching” is powered by result persistence. With a Docker worker, you control where cached results are stored by configuring result storage. You have two main choices: A) Use local storage inside the container (mount a volume for persistence) - Set these environment variables for the container (via your Docker work pool’s default job variables or per-deployment):
Copy code
PREFECT_RESULTS_PERSIST_BY_DEFAULT=true
PREFECT_RESULTS_LOCAL_STORAGE_PATH=/prefect-storage
- Mount a volume to persist across runs:
Copy code
volumes:
  - "/host/path/prefect-storage:/prefect-storage"
- Now, when tasks/flows have 
persist_result=True
(or the global default above), Prefect stores results under 
/prefect-storage
in the container. Without a volume, that cache is ephemeral and will be lost when the container exits. B) Use a remote storage block (recommended for shared/persistent cache across workers) - Create a filesystem block and set it as the default: - S3: 
s3-bucket/my-bucket
- GCS: 
gcs-bucket/my-bucket
- Azure: 
azure-container/my-container
- LocalFileSystem (mounted path on worker): 
local-file-system/my-local-block
- Make it the default for all tasks/flows:
Copy code
PREFECT_RESULTS_PERSIST_BY_DEFAULT=true
PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK=s3-bucket/prod-results
# modern canonical name is PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK
- Or set at the flow/task level:
Copy code
@flow(result_storage="s3-bucket/prod-results", persist_result=True)
def my_flow(): ...

@task(result_storage="gcs-bucket/my-cache", persist_result=True)
def my_task(...): ...
Quick examples - Work pool defaults (UI: Work Pools -> your Docker pool -> Edit -> Default job variables):
Copy code
env:
  PREFECT_RESULTS_PERSIST_BY_DEFAULT: "true"
  PREFECT_RESULTS_LOCAL_STORAGE_PATH: "/prefect-storage"
volumes:
  - "/host/prefect-storage:/prefect-storage"
- Per-deployment job variables:
Copy code
job_variables:
  env:
    PREFECT_RESULTS_PERSIST_BY_DEFAULT: "true"
    PREFECT_RESULTS_LOCAL_STORAGE_PATH: "/prefect-storage"
  volumes:
    - "/host/prefect-storage:/prefect-storage"
- Flow-level remote storage:
Copy code
from prefect import flow, task

@task(persist_result=True, result_storage="gcs-bucket/my-cache")
def compute(x): return x * 2

@flow(persist_result=True, result_storage="gcs-bucket/my-cache")
def pipeline():
    return compute(21)
Key points - Use PREFECT_RESULTS_LOCAL_STORAGE_PATH to control the local cache directory inside the container (modern name; PREFECT_LOCAL_STORAGE_PATH also works but is legacy). - Ensure persistence by either mounting a volume or using a remote storage block; otherwise results are lost when the container exits. - Tasks inherit the flow’s result_storage unless overridden. - You can also control serialization with 
result_serializer
if needed. If you share whether you prefer local volume vs GCS/S3 for caching, I can give you a tailored template for your Docker work pool or deployment.
r
@Marvin - If I want to clear a cache for a flow run, how can i do this?
m
✅ thought for 128.2 seconds
You’ve got a few good options in Prefect 3.x depending on whether you want to bust cache just for this run, for all tasks globally, or by deleting stored results. If you want to ignore cache for the next run (recommended) - One-off for all tasks in the run: set the env var before starting the run so tasks re-execute and overwrite any cached result
Copy code
PREFECT_TASKS_REFRESH_CACHE=true
- Per-task toggle in code:
Copy code
from prefect import task

@task(refresh_cache=True)
def my_task(...):
    ...
This bypasses reading the cache and writes a fresh result back. If you want to disable caching entirely - Global off switch (no read, no write):
Copy code
PREFECT_TASKS_DISABLE_CACHING=true
- Or default to no cache unless tasks opt in:
Copy code
PREFECT_TASKS_DEFAULT_NO_CACHE=true
If you want to delete the existing cached results - Local storage (default): delete files under your results directory - Default path: 
~/.prefect/storage
(inside the container for Docker runs unless you mounted a volume) - If you set a custom path: check 
PREFECT_RESULTS_LOCAL_STORAGE_PATH
- Remove specific entries by filename (cache key) or clear the folder entirely:
Copy code
rm -rf ~/.prefect/storage/*
- Remote storage (S3/GCS/etc.): delete the object(s) at the result keys you used. For example, with S3:
Copy code
aws s3 rm <s3://my-bucket/path/to/cached-object>
Tips for Docker workers - To bust cache for a single flow run executed by a Docker worker, set the env var in the job variables or pool defaults:
Copy code
env:
  PREFECT_TASKS_REFRESH_CACHE: "true"
- If you rely on local on-disk caching, mount a volume so you can inspect/clear it between runs:
Copy code
volumes:
  - "/host/cache:/prefect-cache"
env:
  PREFECT_RESULTS_LOCAL_STORAGE_PATH: "/prefect-cache"
Tell me if you prefer a one-off refresh for a specific deployment or a permanent change, and whether you’re using local or remote result storage—I can provide exact commands/snippets for your setup.
18 Views
Open in Slack
PreviousNext
Powered by