https://prefect.io logo
Join Slack
Powered by
Hi <@ULVA73B9P> i want to override the image being...
# ask-community
i
Hi @Marvin i want to override the image being used for running jobs. My worker has this base image prefecthq/prefect:3.1.15-python3.10-kubernetes However, i have a custom image built on top of this that my job runs should be using
m
✅ thought for 280.6 seconds
Great question — and good news: the image your Kubernetes worker runs with does not have to be the same image your flow runs (jobs) use. You can point your flow runs to your custom image. You have two good ways to set the job image in Prefect 3.x with the Kubernetes worker: Option A — Set a default image on the work pool (applies to all deployments in the pool) - In the UI: Work Pools -> select your Kubernetes pool -> set the Image field to your custom image (e.g., `my-registry.com/my-team/my-image:tag`) - Via CLI: 1) Export and edit the default template
Copy code
prefect work-pool get-default-base-job-template --type kubernetes > template.json
2) In 
template.json
, set:
Copy code
"variables": {
    "image": {
      "default": "<http://my-registry.com/my-team/my-image:tag|my-registry.com/my-team/my-image:tag>"
    }
  }
3) Apply it to your pool
Copy code
prefect work-pool update "YOUR_POOL_NAME" --base-job-template template.json
Option B — Override per deployment (recommended if you only want some jobs to use the custom image) - Using prefect.yaml:
Copy code
deployments:
  - name: my-deployment
    entrypoint: flows/your_flow.py:your_flow
    work_pool:
      name: YOUR_POOL_NAME
      job_variables:
        image: "<http://my-registry.com/my-team/my-image:tag|my-registry.com/my-team/my-image:tag>"
        # optional: image_pull_policy: "Always"
Then deploy:
Copy code
prefect deploy -n my-deployment
- Using Python:
Copy code
from prefect import flow

  @flow
  def your_flow():
      ...

  your_flow.from_source(
      source="<https://github.com/your-org/your-repo.git>",
      entrypoint="flows/your_flow.py:your_flow"
  ).deploy(
      name="my-deployment",
      work_pool_name="YOUR_POOL_NAME",
      job_variables={
          "image": "<http://my-registry.com/my-team/my-image:tag|my-registry.com/my-team/my-image:tag>",
          # optional: "image_pull_policy": "Always",
      },
  )
Precedence - Deployment job_variables override the work pool default. If neither is set, Prefect falls back to the default Prefect image. Private registries - If your image is in a private registry, add imagePullSecrets to the base job template (via Work Pool base job template JSON). See the “Customize base job templates” guide below. Helpful docs - Kubernetes worker/infrastructure: Run flows on Kubernetes - Customize base job templates: Customize base job templates - Customize job variables (per-deployment overrides): Customize job variables - prefect-kubernetes integration overview: prefect-kubernetes integration Want me to help you set this as a pool-wide default or just for specific deployments? If you share your pool name and the exact image (and whether it’s private), I can give you the exact snippet/command to use.
i
@Marvin where to setup custom dependencies for a deployment, will that also go in deployment job variables ?
m
✅ thought for 139.5 seconds
Short answer: no — dependencies aren’t a “deployment job variable.” In Prefect 3.x you generally manage Python deps via your image or build steps. You can use 
job_variables
to pass an env var for runtime installs, but that’s optional. Your options: 1) Bake deps into the image (recommended for production) - If you already have a custom image on top of 
prefecthq/prefect:...
, install your deps there. Then point the deployment to that image via 
job_variables.image
. - With prefect.yaml auto-build (picks up requirements.txt):
Copy code
build:
    - prefect_docker.deployments.steps.build_docker_image:
        id: build-image
        requires: prefect-docker>=0.4.0
        image_name: my-registry/my-image
        tag: latest
        dockerfile: auto   # auto Dockerfile installs requirements.txt

  deployments:
    - name: my-deployment
      entrypoint: flows/my_flow.py:my_flow
      work_pool:
        name: kubernetes
        job_variables:
          image: "{{ build-image.image }}"
2) Quick runtime installs via job variables (good for dev/testing) - Set 
EXTRA_PIP_PACKAGES
under 
env
in `job_variables`:
Copy code
deployments:
    - name: my-deployment
      entrypoint: flows/my_flow.py:my_flow
      work_pool:
        name: kubernetes
        job_variables:
          env:
            EXTRA_PIP_PACKAGES: "pandas==2.2.2 boto3>=1.28.0"
- The worker will run 
uv pip install
for those packages at job start (adds overhead each run). 3) Custom Dockerfile with Python API (advanced/private indexes, build args) - Use 
DockerImage
in 
.deploy()
to control the build exactly (Dockerfile, build args, private registries). Docs to dive deeper: - Customize job variables: Customize job variables - Deploy with prefect.yaml (build/push/deploy): prefect.yaml deployments - Deploy via Python (using `image`/`DockerImage`): Deploy via Python - Docker/Kubernetes infra guides: Run flows with Docker and Run flows on Kubernetes If you share how you prefer to manage deps (prebaked image vs. runtime installs) and where your code lives (local vs. Git), I can give you a minimal, exact snippet for your setup.
i
@Marvin when i use a custom image for my flow runs , i get this error Unable to connect to 'ws://prefect-server-external.prefect.svc.cluster.local:4200/api/events/in'. Please check your network settings to ensure websocket connections to the API are allowed. Otherwise event data (including task run data) may be lost. Reason: Unable to authenticate to the event stream. Please ensure the provided auth_token you are using is valid for this environment.
m
✅ thought for 187.7 seconds
Thanks for the details. That error means your flow-run container connected to your Prefect Server’s event stream but failed the auth handshake. With a custom image, the most common causes are: - Missing/wrong PREFECT_API_AUTH_STRING in the flow-run container (for self-hosted) - PREFECT_API_KEY is set (Cloud-only) and overrides auth_string - The work pool’s base job template doesn’t pass Prefect settings into the pod (env injection removed/overridden) - Less common: version mismatch or TLS/WS configuration Quick fixes to try first 1) If you’re self-hosting Prefect Server and it has an auth string configured, set the same value in your flow-run containers: - On the server, confirm it’s set:
Copy code
echo $PREFECT_SERVER_API_AUTH_STRING
- In your deployment (via job variables), pass it down:
Copy code
work_pool:
  name: YOUR_POOL
  job_variables:
    env:
      PREFECT_API_URL: "<http://prefect-server-external.prefect.svc.cluster.local:4200/api>"
      PREFECT_API_AUTH_STRING: "admin:password"  # match the server’s value
Note: For production, use a Kubernetes Secret and a customized base job template with valueFrom; job_variables.env only supports plain strings. 2) Make sure PREFECT_API_KEY is NOT set in the flow-run container - API keys are for Prefect Cloud. If both are present, API_KEY takes precedence and self-hosted auth will fail. Remove it from your image/env:
Copy code
unset PREFECT_API_KEY
3) Verify your work pool’s base job template includes env passthrough - The container spec must include 
"env": "{{ env }}"
. If you removed or replaced this, Prefect settings won’t be injected into the pod. - Check the template:
Copy code
prefect work-pool inspect YOUR_POOL
prefect work-pool get-default-base-job-template --type kubernetes
If you customized it, ensure the pod/container has 
"env": "{{ env }}"
so Prefect settings (including PREFECT_API_URL and PREFECT_API_AUTH_STRING) are injected. 4) Inspect a failing pod’s environment - Confirm the pod actually has the right settings:
Copy code
kubectl -n <ns> describe pod <pod-name> | sed -n '/Environment:/,/Mounts:/p'
Look for: - PREFECT_API_URL = http://prefect-server-external.prefect.svc.cluster.local:4200/api - PREFECT_API_AUTH_STRING = your “user:password” (if server uses auth) - Ensure PREFECT_API_KEY is not present 5) Version/TLS checks (less likely here) - Keep client (your custom image’s Prefect version) and server reasonably aligned. - If you use TLS, use WSS and set these as needed:
Copy code
PREFECT_API_TLS_INSECURE_SKIP_VERIFY=true
PREFECT_API_SSL_CERT_FILE=/path/to/ca.pem
Why this happens with a custom image - The image change doesn’t affect network reachability (you’re connecting) but can change what env vars exist in the container. If PREFECT_API_AUTH_STRING isn’t set (or PREFECT_API_KEY is set mistakenly), the event-stream handshake fails. If you want a quick test - Add this to your deployment’s job_variables temporarily, redeploy, and re-run:
Copy code
job_variables:
  env:
    PREFECT_API_URL: "<http://prefect-server-external.prefect.svc.cluster.local:4200/api>"
    PREFECT_API_AUTH_STRING: "admin:password"
If logs start streaming, move this into a more secure setup (K8s Secret + base job template with valueFrom). Docs - Customize job variables (env overrides): Customize job variables - Customize base job templates (add imagePullSecrets, valueFrom, etc.): Customize base job templates - Run flows on Kubernetes: Run flows on Kubernetes Can you confirm: - Are you using self-hosted Prefect Server (not Cloud)? - Is PREFECT_SERVER_API_AUTH_STRING set on your server? - Have you customized the work pool’s base job template? - What do you see for PREFECT_API_* env vars inside a failing pod?
i
@Marvin but my prefect version doesn't have any auth configured, why is it asking for an auth string ? And failing
m
✅ thought for 76.7 seconds
Got it — Prefect Server only requires an auth string if it was configured with one. If you’re seeing “Unable to authenticate to the event stream,” one of these is usually true: - The server actually has PREFECT_SERVER_API_AUTH_STRING set (sometimes via Helm/values), even if you didn’t intend to - Your flow-run pod isn’t getting Prefect settings injected (custom base job template removed 
"env": "{{ env }}"
) - You’re not actually talking to the server you think you are (wrong URL/ingress), or there’s a version mismatch causing the handshake to be treated like an auth failure Here’s how to pinpoint it quickly: 1) Confirm what server you’re hitting and whether it expects auth - From a failing flow-run pod:
Copy code
# See where the SDK is connecting
env | grep PREFECT_API_URL

# Optional: quick reachability check (will not “work” but should show no 401)
curl -i <http://prefect-server-external.prefect.svc.cluster.local:4200/api/events/in>
- On the Prefect Server pod:
Copy code
# If this prints anything non-empty, auth is enabled on the server
printenv PREFECT_SERVER_API_AUTH_STRING
If that var is set, either unset it on the server or set the matching client var in your flow-run pods:
Copy code
PREFECT_API_AUTH_STRING="user:password"
2) Make sure your pod actually has Prefect settings injected - If your work pool base job template was customized, verify the container spec includes:
Copy code
"env": "{{ env }}"
Without that, the pod won’t inherit Prefect settings (including API URL/auth). - Inspect your pool and compare with the default:
Copy code
prefect work-pool inspect YOUR_POOL
prefect work-pool get-default-base-job-template --type kubernetes
- You can also check a failing pod:
Copy code
kubectl -n <ns> describe pod <pod> | sed -n '/Environment:/,/Mounts:/p'
3) Rule out version/endpoint mismatch - What Prefect version is your server? If you’re running a Prefect 2.x server with 3.x workers/flows, the event stream endpoint/handshake won’t match and can look like an auth failure. If that’s the case, upgrade the server to 3.x or align versions. - You can also enable debug logs for more detail:
Copy code
PREFECT_DEBUG_MODE=1
If you need a quick test to confirm it’s purely auth/string-related, add this temporarily to your deployment:
Copy code
work_pool:
  name: YOUR_POOL
  job_variables:
    env:
      PREFECT_API_URL: "<http://prefect-server-external.prefect.svc.cluster.local:4200/api>"
      PREFECT_API_AUTH_STRING: "user:password"  # only if server actually has it set
If that resolves it, move the value into a Kubernetes Secret and reference it via a customized base job template (valueFrom). Helpful references (for env passthrough and templates): - Customize base job templates (ensure 
"env": "{{ env }}"
is present): Customize base job templates - Kubernetes worker/infrastructure: Run flows on Kubernetes - Customize job variables (for temporary env overrides): Customize job variables If you can share: - Output of 
printenv PREFECT_SERVER_API_AUTH_STRING
from the server pod - Prefect version of your server - A snippet of your work pool base job template (the 
containers[0]
section) …I can tell you exactly which fix applies.
i
@Marvin this is null for my 
PREFECT_SERVER_API_AUTH_STRING
server i dont understand why its asking for an auth token this is my custom docker image that i am using in as my job variable FROM prefecthq/prefect:3.1.15-python3.10 # Define common paths as build args and env vars ARG DBT_VENV="/home/ddp/dbt" ARG CLIENTDBT_ROOT="/mnt/appdata/clientdbts" ENV DBT_VENV=${DBT_VENV} ENV CLIENTDBT_ROOT=${CLIENTDBT_ROOT} # Install system dependencies RUN apt-get update && apt-get install -y \ python3-venv \ python3-dev \ git \ build-essential \ curl \ && rm -rf /var/lib/apt/lists/* # Install additional Prefect packages RUN pip install git+https://github.com/Ishankoradia/prefect-airbyte.git@v0.90 prefect-dbt==0.6.6 # Create the exact directory structure matching your current setup RUN mkdir -p ${DBT_VENV} \ && mkdir -p ${CLIENTDBT_ROOT} # Copy dbt version directories with their lock files COPY dbt-1.9.8/ /tmp/dbt-1.9.8/ COPY dbt-1.8.7/ /tmp/dbt-1.8.7/ COPY dbt-1.10.19/ /tmp/dbt-1.10.19/ # Create dbt 1.8.7 venv (default - matches your current venv path) RUN python3 -m venv /home/ddp/dbt/venv && \ cd /tmp/dbt-1.8.7 && \ /home/ddp/dbt/venv/bin/pip install --no-cache-dir . && \ rm -rf /tmp/dbt-1.8.7 # Create dbt 1.9.8 venv RUN python3 -m venv /home/ddp/dbt/venv-1.9.8 && \ cd /tmp/dbt-1.9.8 && \ /home/ddp/dbt/venv-1.9.8/bin/pip install --no-cache-dir . && \ rm -rf /tmp/dbt-1.9.8 # Create dbt 1.10.19 venv RUN python3 -m venv /home/ddp/dbt/venv-1.10.19 && \ cd /tmp/dbt-1.10.19 && \ /home/ddp/dbt/venv-1.10.19/bin/pip install --no-cache-dir . && \ rm -rf /tmp/dbt-1.10.19 # Final cleanup - remove build tools and any remaining artifacts RUN apt-get purge -y build-essential python3-dev curl && \ apt-get autoremove -y && \ rm -rf /var/lib/apt/lists/* /root/.cache /tmp/* /var/tmp/*
m
✅ thought for 234.6 seconds
Thanks for sharing the details and the Dockerfile. If 
PREFECT_SERVER_API_AUTH_STRING
is null on your server, Prefect Server itself is not requiring Basic auth. In that case, this “Unable to authenticate to the event stream” usually points to one of these: - Your flow-run pod is talking to an endpoint that adds its own auth (Ingress/external Service), not the bare Server API - A Cloud-style API key is present in the pod and the client is attempting the wrong auth scheme - Env injection in the job template is off, or there’s a version/endpoint mismatch Given your ws URL is: ws://prefect-server-external.prefect.svc.cluster.local:4200/api/events/in That “-external” service name is a strong hint: you’re likely hitting an external/ingress-facing Service that may be enforcing auth or rewriting WebSocket traffic. Inside the cluster, you should target the internal ClusterIP Service for the API to avoid extra auth and WS proxying issues. What to try first 1) Point runs to the internal service - Set 
PREFECT_API_URL
to the internal service name (adjust namespace/name as needed):
Copy code
<http://prefect-server.prefect.svc.cluster.local:4200/api>
- Add it via deployment job variables or in the work pool base job template so it flows into the pod:
Copy code
work_pool:
  name: YOUR_POOL
  job_variables:
    env:
      PREFECT_API_URL: "<http://prefect-server.prefect.svc.cluster.local:4200/api>"
2) Ensure no Cloud API key is present - Inside a failing pod, check:
Copy code
env | grep PREFECT_API_
Confirm: - PREFECT_API_KEY is NOT set (Cloud-only; if present it can trigger the wrong auth path) - PREFECT_API_AUTH_STRING is empty (fine for self-hosted without auth) - PREFECT_API_URL points to the internal service as above 3) Verify env passthrough in your work pool template - Your container spec should include:
Copy code
"env": "{{ env }}"
If it’s missing/overridden, Prefect settings won’t reach the pod. Compare your pool to the default:
Copy code
prefect work-pool inspect YOUR_POOL
prefect work-pool get-default-base-job-template --type kubernetes
4) Quick connectivity check from the pod - These commands won’t fully “work” but will show status codes (401/403 here would confirm an auth layer on the external path):
Copy code
curl -i <http://prefect-server-external.prefect.svc.cluster.local:4200/api/health>
curl -i <http://prefect-server-external.prefect.svc.cluster.local:4200/api/events/in>
curl -i <http://prefect-server.prefect.svc.cluster.local:4200/api/health>
5) Keep versions aligned - Check server and custom image Prefect versions; try to keep them close (you’re on 3.1.15 in the image). If your server is older/newer by a lot, align them. Your Dockerfile looks fine with respect to Prefect networking/auth; nothing there would trigger this specific error. If you can share: - The output of 
env | grep PREFECT_API_
from a failing pod - The Kubernetes work pool base job template 
containers[0]
snippet (to confirm 
"env": "{{ env }}"
) - Your server Service names/namespace (so we can confirm the correct internal service URL) …I can give you an exact 
job_variables
or pool template patch. For reference: - Customize base job templates: Customize base job templates - Customize job variables (env overrides): Customize job variables - Run flows on Kubernetes: Run flows on Kubernetes
i
@Marvin i am at prefect 3.1.15 versions I have a custom docker image, no auth is configured Why does my flow run pod says this 131825.764 | DEBUG | prefect.client - Connecting to API at http://prefect-server-external.prefect.svc.cluster.local:4200/api/ 131825.765 | DEBUG | prefect.events.clients - Reconnecting websocket connection. 131825.765 | DEBUG | prefect.events.clients - Opening websocket connection. 131825.770 | DEBUG | prefect.events.clients - Pinging to ensure websocket connected. 131825.771 | DEBUG | prefect.events.clients - Pong received. Websocket connected. 131825.771 | DEBUG | prefect.events.clients - Authenticating... 131825.776 | WARNING | prefect.events.clients - Unable to connect to 'ws://prefect-server-external.prefect.svc.cluster.local:4200/api/events/in'. Please check your network settings to ensure websocket connections to the API are allowed. Otherwise event data (including task run data) may be lost. Reason: Unable to authenticate to the event stream. Please ensure the provided auth_token you are using is valid for this environment. . Set PREFECT_DEBUG_MODE=1 to see the full error. Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/prefect/events/clients.py", line 374, in _auth_handshake message: Dict[str, Any] = orjson.loads(await self._websocket.recv()) File "/usr/local/lib/python3.10/site-packages/websockets/asyncio/connection.py", line 324, in recv raise self.protocol.close_exc from self.recv_exc websockets.exceptions.ConnectionClosedError: no close frame received or sent The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/prefect/events/clients.py", line 305, in aenter await self._reconnect() File "/usr/local/lib/python3.10/site-packages/prefect/events/clients.py", line 417, in _reconnect await self._auth_handshake() File "/usr/local/lib/python3.10/site-packages/prefect/events/clients.py", line 388, in _auth_handshake raise Exception(msg) from e Exception: Unable to authenticate to the event stream. Please ensure the provided auth_token you are using is valid for this environment. Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/prefect/events/clients.py", line 374, in _auth_handshake message: Dict[str, Any] = orjson.loads(await self._websocket.recv()) File "/usr/local/lib/python3.10/site-packages/websockets/asyncio/connection.py", line 324, in recv raise self.protocol.close_exc from self.recv_exc websockets.exceptions.ConnectionClosedError: no close frame received or sent The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/prefect/cli/_utilities.py", line 37, in async_wrapper return await fn(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/prefect/cli/flow_run.py", line 627, in execute await runner.execute_flow_run(id) File "/usr/local/lib/python3.10/site-packages/prefect/runner/runner.py", line 682, in execute_flow_run async with context: File "/usr/local/lib/python3.10/site-packages/prefect/runner/runner.py", line 1682, in aenter await self._exit_stack.enter_async_context(self._events_client) File "/usr/local/lib/python3.10/contextlib.py", line 619, in enter_async_context result = await _cm_type.__aenter__(cm) File "/usr/local/lib/python3.10/site-packages/prefect/events/clients.py", line 305, in aenter await self._reconnect() File "/usr/local/lib/python3.10/site-packages/prefect/events/clients.py", line 417, in _reconnect await self._auth_handshake() File "/usr/local/lib/python3.10/site-packages/prefect/events/clients.py", line 388, in _auth_handshake raise Exception(msg) from e Exception: Unable to authenticate to the event stream. Please ensure the provided auth_token you are using is valid for this environment. An exception occurred.
m
✅ thought for 54.7 seconds
Thanks for the detailed logs. On 3.1.15 with “no auth configured,” that message is almost always a symptom of the websocket being routed to something that is not the Prefect API’s event stream (e.g., an ingress/external service that closes the connection). The client wraps any early-close during the handshake as “Unable to authenticate…”, which is misleading. In your logs, the client is connecting to: ws://prefect-server-external.prefect.svc.cluster.local:4200/api/events/in The “-external” service name is a red flag. Inside the cluster, that often points to an ingress/edge path which may: - enforce auth or other middleware, - not route /api/events/in to the Prefect API, - or terminate websockets differently. Recommended fix - Point your flow runs at the internal ClusterIP service for the Prefect API instead of the external service. For many installs this is:
Copy code
PREFECT_API_URL=<http://prefect-server.prefect.svc.cluster.local:4200/api>
Set it on your deployment (or make it pool-wide) so it flows into the pod:
Copy code
work_pool:
  name: YOUR_POOL
  job_variables:
    env:
      PREFECT_API_URL: "<http://prefect-server.prefect.svc.cluster.local:4200/api>"
Then verify inside a failing pod:
Copy code
env | grep PREFECT_API_
# Expect:
# PREFECT_API_URL=<http://prefect-server.prefect.svc.cluster.local:4200/api>
# PREFECT_API_KEY not set
# PREFECT_API_AUTH_STRING not set (fine, since server has none)
Also double-check your work pool base job template still passes env through:
Copy code
prefect work-pool get-default-base-job-template --type kubernetes
# In containers[0], ensure you have:
# "env": "{{ env }}"
If that’s missing in a customized template, Prefect settings won’t reach the pod. If you must use the external/ingress route - Ensure your ingress routes /api/events/in to the Prefect API service and allows websocket upgrades on that path (no extra auth). For NGINX, confirm websocket support and timeouts; for Traefik/other gateways, ensure Upgrade/Connection headers are preserved. Other sanity checks - Keep server and image Prefect versions close (you’re on 3.1.15 in the image; match the server if possible). - Server-side event streaming is enabled by default; if it was disabled, the server could close the socket early. You can confirm on the server pod:
Copy code
printenv PREFECT_SERVER_EVENTS_STREAM_OUT_ENABLED
# should be empty or "true"
If you share: - The Service names/namespace of your Prefect API, - The output of 
env | grep PREFECT_API_
from a failing flow-run pod, - The 
containers[0]
section of your pool’s base job template, …I’ll give you the exact 
job_variables
or template patch to apply. Helpful refs: - Run flows on Kubernetes - Customize base job templates - Customize job variables
12 Views
Open in Slack
PreviousNext
Powered by