https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> When I submit a flow and job has fail...
# marvin-ai
k
@Marvin When I submit a flow and job has failed to be scheduled due to various reasons so the job is stuck in pending. When I try to cancel the flow on the dashboard it's then stuck in "cancelling" and the job is not terminated. how to avoid this?
m
✅ thought for 166.3 seconds
This usually means Prefect knows the flow run should be cancelled, but it either: 1. never got far enough to create/track the underlying infrastructure job, or 2. cannot reach the worker/infrastructure to kill it, or 3. the run is in a gap state like Pending where cancellation cleanup does not complete cleanly. What’s happening under the hood In Prefect 3.x, cancelling a run from the UI typically moves it to Cancelling first, and then a worker is expected to: - find the associated infrastructure job - terminate it - mark the flow run Cancelled If the job failed to schedule and the run is stuck in Pending, there may be no infrastructure PID/job handle for the worker to kill. In that case, the flow run can get stranded in Cancelling. Common reasons this happens - The worker submitted the run, but infrastructure creation failed before a job ID / infrastructure PID was recorded - The worker is offline or disconnected when cancellation happens - The worker type cannot successfully execute 
kill_infrastructure()
- The underlying platform is unavailable (Kubernetes API, Docker daemon, cloud batch service, etc.) - The run is in Pending rather than Running/Scheduled, so there is effectively nothing concrete to terminate How to avoid it 1. Make sure workers stay healthy and connected - If the worker is down when you cancel, cleanup may never happen. - Check worker health/heartbeat and logs around the time the run entered Pending or Cancelling. 2. Investigate why jobs fail to schedule in the first place This is the most important fix. If scheduling succeeds reliably, cancellation is usually straightforward. Look for worker/infrastructure errors such as: - bad image / image pull errors - missing Kubernetes service account / RBAC / quota - invalid job variables - missing Docker daemon access - cloud batch/API permissions - namespace / resource constraints 3. Use work pools/workers that support infrastructure killing properly If your worker type cannot terminate infrastructure cleanly, runs may remain in Cancelling. 4. Add an operational cleanup path for stuck Cancelling runs If a run has been in Cancelling for a long time and the job never actually started, force it to Cancelled. Example using the Prefect client:
Copy code
python
import asyncio
from uuid import UUID
from prefect import get_client
from prefect.states import Cancelled

FLOW_RUN_ID = UUID("YOUR-FLOW-RUN-ID")

async def force_cancel():
    async with get_client() as client:
        await client.set_flow_run_state(
            FLOW_RUN_ID,
            Cancelled(message="Force cancelled after stuck in Cancelling"),
            force=True,
        )

asyncio.run(force_cancel())
That does not terminate external infrastructure by itself — it just fixes the Prefect state if there is no real job left to kill. 5. Monitor for stale Pending/Cancelling runs A practical pattern is to periodically: - find runs stuck in Pending or Cancelling beyond a threshold - inspect whether infrastructure was ever created - if not, force them to Cancelled What I’d check next - Worker logs for the specific run - Whether the flow run has an 
infrastructure_pid
- Whether the underlying job actually exists in your execution platform - Whether the worker was online when you pressed cancel Important nuance If the job never scheduled successfully, there may be nothing to terminate. In that case, the fix is not really “kill harder” — it’s: - resolve the infrastructure submission failure - and optionally force-close orphaned Prefect states Relevant references - Prefect deployments/workers docs: Deployments and workers - Work pools docs: Work pools - How workers execute flow runs: Run flows with workers
If you want, I can help you narrow this down for your setup specifically — e.g. Kubernetes worker, Docker worker, process worker, or Cloud Run / ECS / ACI — since the exact failure mode and best prevention depends a lot on the worker type.
k
@Marvin It's kubernetes worker
m
✅ thought for 174.8 seconds
With a Kubernetes worker, the key distinction is: - Job created successfully != Pod actually scheduled/running successfully So you can end up with: - Prefect flow run in Pending - Kubernetes 
Job
exists - Pod is stuck 
Pending
or never created correctly - You click Cancel - Prefect moves to Cancelling - worker cannot cleanly delete the K8s job/pod or never gets a clean response - run remains stuck in Cancelling Most common Kubernetes-specific causes 1. Pod unschedulable - insufficient CPU/memory - node selector / affinity mismatch - missing tolerations for tainted nodes 2. Image pull problems - bad image tag - private registry auth missing - 
ImagePullBackOff
/ 
ErrImagePull
3. Namespace / RBAC issues - worker can create/read jobs but cannot delete them - worker service account lacks permission in the target namespace 4. Resource quotas / limits - namespace quota exceeded - LimitRange/defaults conflict with requested resources 5. Job exists but no runnable pod - job deletion may stall or fail from the worker side - Prefect state gets stuck waiting for cleanup What to check first For a stuck run, inspect all three layers: 1. Prefect run details Check whether the run has an 
infrastructure_pid
. For Kubernetes workers this is typically effectively 
namespace:job_name
. If there is no 
infrastructure_pid
, then Prefect has nothing concrete to kill, and cancellation often needs manual state repair. 2. Kubernetes Job Check whether the job exists:
Copy code
bash
kubectl get jobs -n <namespace>
kubectl describe job <job-name> -n <namespace>
3. Kubernetes Pod/events If the job exists, inspect pod scheduling:
Copy code
bash
kubectl get pods -n <namespace> -l job-name=<job-name>
kubectl describe pod <pod-name> -n <namespace>
Look at the Events section for: - 
FailedScheduling
- 
ImagePullBackOff
- 
ErrImagePull
- 
Forbidden
- 
Exceeded quota
How to avoid this going forward 1. Reduce unschedulable pods Make sure your work pool job template does not request unrealistic resources. Typical things to review: - CPU/memory requests and limits - node selectors / affinities - tolerations - namespace quotas 2. Verify worker RBAC includes delete permissions Your worker needs to be able to delete jobs as well as create them. At minimum, test:
Copy code
bash
kubectl auth can-i create jobs --as=system:serviceaccount:<ns>:<sa> -n <namespace>
kubectl auth can-i delete jobs --as=system:serviceaccount:<ns>:<sa> -n <namespace>
kubectl auth can-i get pods --as=system:serviceaccount:<ns>:<sa> -n <namespace>
kubectl auth can-i list pods --as=system:serviceaccount:<ns>:<sa> -n <namespace>
If delete is missing, cancellation can absolutely get stuck. 3. Validate image pullability Before using an image in the deployment: - confirm the tag exists - confirm nodes can access the registry - confirm 
imagePullSecrets
if needed 4. Enable better worker logging Run the worker with debug logging so you can see whether: - job creation succeeded - 
infrastructure_pid
was recorded - delete job call failed - Kubernetes API returned 403/404/timeout 5. Treat “Pending too long” as an operational alert If a run is 
Pending
for more than a few minutes, that’s usually a cluster/config problem, not a Prefect scheduling delay. Operational recovery when one is already stuck If the job exists, delete it manually:
Copy code
bash
kubectl delete job <job-name> -n <namespace>
If that hangs or the pod is wedged, force it:
Copy code
bash
kubectl delete job <job-name> -n <namespace> --grace-period=0 --force
kubectl delete pods -n <namespace> -l job-name=<job-name> --grace-period=0 --force
Then force the Prefect flow run state to Cancelled. Using Python client: ```python import asyncio from uuid import UUID from prefect import get_client from prefect.states import Cancelled FLOW_RUN_ID = UUID("YOUR-FLOW-RUN-ID") async def fix_state():
async with get_client() as client: await client.set_flow_run_state( FLOW_RUN_ID, Cancelled(message="Manually cancelled after Kubernetes job cleanup"), force=True, ) asyncio.run(fix_state()) ``` Practical interpretation - If Job exists: this is mostly a Kubernetes scheduling/deletion issue - If Job does not exist and no `infrastructure_pid`: this is mostly a submission/recording issue, and you’ll likely need to force-close the Prefect state - If worker lacks RBAC to delete jobs: cancellation can keep getting stuck in Cancelling Best prevention checklist - sane resource requests/limits - correct namespace - worker service account can 
create/get/delete/list
jobs and inspect pods - image pull secrets configured - monitor 
FailedScheduling
and 
ImagePullBackOff
- alert on runs stuck in 
Pending
or 
Cancelling
Useful docs - Kubernetes worker docs: Run flows on Kubernetes - Work pools: Work pools - Deployments/workers overview: Deployments and workers If you want, I can help you with a concrete debug checklist for your cluster, or help you inspect your Kubernetes work pool job template for the exact fields that commonly cause this.
k
@Marvin How can I have two wokers running in two different clusters pointing to the same workpool?
m
✅ thought for 166.6 seconds
Yes — you can have two Kubernetes workers in two different clusters polling the same work pool in Prefect 3.x. But there’s an important caveat: If both workers poll the same pool/queue, Prefect does not route by cluster. Whoever grabs the run first will execute it. So the answer is: - Yes, same work pool is supported - But if you need deterministic cluster placement, use different work queues in that same pool Recommended pattern Use: - one Kubernetes work pool - one queue per cluster - one worker per cluster polling only its queue That gives you: - shared administration in one work pool - deterministic routing - easier debugging/cancellation How it works If you do this: - Cluster A worker polls 
my-pool
- Cluster B worker polls 
my-pool
then without queue separation: - any scheduled run in that pool can be picked up by either worker - it’s basically first-come-first-served If instead you do this: - Cluster A worker polls queue 
cluster-a
- Cluster B worker polls queue 
cluster-b
then deployments targeted at 
cluster-a
will only be picked up by the worker in cluster A. Best practice setup 1. Create one Kubernetes work pool 2. Create a queue for each cluster - 
cluster-a
- 
cluster-b
3. Run one worker in each cluster, each polling only its own queue 4. Point deployments to the correct queue This is the cleanest multi-cluster setup. Worker commands I wasn’t able to verify the CLI help live because the check timed out, so I’ll avoid overcommitting to flags beyond the standard pattern. Typical startup pattern is:
Copy code
bash
prefect worker start --pool my-k8s-pool --type kubernetes --name worker-cluster-a --work-queue cluster-a
and in the other cluster:
Copy code
bash
prefect worker start --pool my-k8s-pool --type kubernetes --name worker-cluster-b --work-queue cluster-b
If your installed CLI prefers 
-q
/ 
--work-queue
, use that form from 
prefect worker start --help
in your environment. Deployment routing example In 
prefect.yaml
, target the queue:
Copy code
yaml
deployments:
  - name: my-flow-cluster-a
    entrypoint: flows/my_flow.py:my_flow
    work_pool:
      name: my-k8s-pool
      queue_name: cluster-a
And for cluster B:
Copy code
yaml
deployments:
  - name: my-flow-cluster-b
    entrypoint: flows/my_flow.py:my_flow
    work_pool:
      name: my-k8s-pool
      queue_name: cluster-b
Important caveat for Kubernetes workers Each worker launches jobs in its own cluster context unless you explicitly override Kubernetes cluster configuration. So if both workers share the same pool and same queue: - a run intended for cluster A might get launched in cluster B That’s why queues are the important boundary. Cancellation implications This matters for your earlier issue too: - the worker that picked up the run is the one expected to cancel its Kubernetes job - if work is not cluster-separated, debugging and cancellation become much harder - queue-per-cluster makes it much easier to know which worker/cluster owns the job Extra recommendations - Give workers distinct names like 
worker-eks-prod
and 
worker-gke-prod
- Use separate queues per cluster - Optionally pass env vars/job labels so runs can log which cluster they are on - Keep namespaces and RBAC aligned per cluster Suggested architecture - Work pool: 
my-k8s-pool
- Queues: - 
cluster-a
- 
cluster-b
- Workers: - cluster A worker polls 
cluster-a
- cluster B worker polls 
cluster-b
This is usually better than making separate work pools unless the clusters need very different base job templates or governance boundaries. Docs - Work pools: Work pools - Kubernetes deployments/worker usage: Run flows on Kubernetes - Deployment concepts: Deployments and workers
If you want, I can give you a concrete example of: - the queue setup, - the worker Helm values for each cluster, - and a 
prefect.yaml
that routes deployments to cluster A vs cluster B.
k
@Marvin yes, give me the concrete examples. i am deploying worker pool using helm.
m
✅ thought for 25.3 seconds
Absolutely — here’s a concrete Helm + Prefect work pool + queues + deployment example for two Kubernetes clusters sharing one work pool. Recommended design - One work pool: 
multi-cluster-k8s
- Two queues: 
cluster-a
, 
cluster-b
- One Helm-installed worker per cluster - Each worker polls only its own queue This avoids ambiguous routing. --- 1. Create the work pool In Prefect 3.x, create a Kubernetes work pool:
Copy code
bash
prefect work-pool create --type kubernetes multi-cluster-k8s
You can confirm it in the UI or CLI. Docs: Work pools --- 2. Create one queue per cluster Create queues in that pool:
Copy code
bash
prefect work-pool queue create cluster-a --pool multi-cluster-k8s
prefect work-pool queue create cluster-b --pool multi-cluster-k8s
If your installed CLI differs slightly, check:
Copy code
bash
prefect work-pool queue --help
The intent is: - queue 
cluster-a
for worker in cluster A - queue 
cluster-b
for worker in cluster B --- 3. Helm values for worker in cluster A Below is a representative 
values-cluster-a.yaml
. You will need to adapt: - API URL / Cloud config - namespace - image - auth secret references - chart keys if your chart version differs
Copy code
yaml
worker:
  enabled: true
  type: kubernetes

  config:
    workPool: multi-cluster-k8s
    name: worker-cluster-a

  extraEnvVars:
    PREFECT_LOGGING_LEVEL: INFO

  # If supported by your chart/version, restrict worker to a queue
  # Some chart versions expose this as args instead of config.
  args:
    - prefect
    - worker
    - start
    - --pool
    - multi-cluster-k8s
    - --type
    - kubernetes
    - --name
    - worker-cluster-a
    - --work-queue
    - cluster-a

prefect:
  createApiKeySecret: false
  apiKeySecret:
    name: prefect-api-key

  serverApiConfig:
    accountId: "<ACCOUNT_ID>"
    workspaceId: "<WORKSPACE_ID>"

serviceAccount:
  create: true
  name: prefect-worker

rbac:
  create: true
Install in cluster A:
Copy code
bash
helm upgrade --install prefect-worker prefect/prefect-worker \
  -n prefect \
  --create-namespace \
  -f values-cluster-a.yaml
--- 4. Helm values for worker in cluster B `values-cluster-b.yaml`:
Copy code
yaml
worker:
  enabled: true
  type: kubernetes

  config:
    workPool: multi-cluster-k8s
    name: worker-cluster-b

  extraEnvVars:
    PREFECT_LOGGING_LEVEL: INFO

  args:
    - prefect
    - worker
    - start
    - --pool
    - multi-cluster-k8s
    - --type
    - kubernetes
    - --name
    - worker-cluster-b
    - --work-queue
    - cluster-b

prefect:
  createApiKeySecret: false
  apiKeySecret:
    name: prefect-api-key

  serverApiConfig:
    accountId: "<ACCOUNT_ID>"
    workspaceId: "<WORKSPACE_ID>"

serviceAccount:
  create: true
  name: prefect-worker

rbac:
  create: true
Install in cluster B:
Copy code
bash
helm upgrade --install prefect-worker prefect/prefect-worker \
  -n prefect \
  --create-namespace \
  -f values-cluster-b.yaml
--- 5. Kubernetes secret for Prefect API key In each cluster, create the secret referenced above:
Copy code
bash
kubectl create secret generic prefect-api-key \
  -n prefect \
  --from-literal=key='<PREFECT_API_KEY>'
That key should belong to the same Prefect Cloud workspace. Cloud app: Prefect Cloud --- 6. Minimal RBAC example Your worker service account needs to create and delete jobs and inspect pods. Example: ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: prefect-worker namespace: prefect --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: prefect-worker namespace: prefect rules: - apiGroups: ["batch"] resources: ["jobs"] verbs: ["create", "get", "list", "watch", "delete", "patch"] - apiGroups: [""] resources: ["pods", "pods/log"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["get", "list", "watch"]
- apiGroups: [""] resources: ["secrets"] verbs: ["create", "get", "list", "delete"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: prefect-worker namespace: prefect subjects: - kind: ServiceAccount name: prefect-worker namespace: prefect roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: prefect-worker
Copy code
If your jobs run in a different namespace, scope RBAC there too.

---

*7. Example `prefect.yaml` with cluster-specific queue routing*

Here’s a concrete deployment config:

```yaml
name: multi-cluster-demo

deployments:
  - name: flow-on-cluster-a
    entrypoint: flows/hello.py:hello_flow
    work_pool:
      name: multi-cluster-k8s
      queue_name: cluster-a
      job_variables:
        namespace: prefect
        image: my-registry/my-flow-image:latest
        image_pull_policy: IfNotPresent
        finished_job_ttl: 3600
        env:
          EXECUTION_CLUSTER: cluster-a

  - name: flow-on-cluster-b
    entrypoint: flows/hello.py:hello_flow
    work_pool:
      name: multi-cluster-k8s
      queue_name: cluster-b
      job_variables:
        namespace: prefect
        image: my-registry/my-flow-image:latest
        image_pull_policy: IfNotPresent
        finished_job_ttl: 3600
        env:
          EXECUTION_CLUSTER: cluster-b
Then deploy with:
Copy code
bash
prefect deploy
This way: - deployment 
flow-on-cluster-a
only goes to cluster A worker - deployment 
flow-on-cluster-b
only goes to cluster B worker --- 8. Example flow to confirm which cluster executed it `flows/hello.py`:
Copy code
python
import os
from prefect import flow

@flow
def hello_flow():
    cluster = os.getenv("EXECUTION_CLUSTER", "unknown")
    print(f"Hello from {cluster}")
This makes it obvious from logs which cluster ran the flow. --- 9. Operational checks After installation, verify workers are registered: - in Prefect UI under work pool workers - or via CLI if you use it locally You should see distinct worker names: - 
worker-cluster-a
- 
worker-cluster-b
And confirm in Kubernetes:
Copy code
bash
kubectl get pods -n prefect
kubectl logs -n prefect deploy/prefect-worker
Do this in both clusters. --- 10. What if you want “either cluster can run it”? You can point both workers at the same queue, but I would not recommend that for your setup given the cancellation issues you described earlier. If you do: - routing is non-deterministic - either cluster may pick the run - cancellation ownership is less obvious Much better: - use one queue per cluster for deterministic placement - optionally create a third shared queue only for workloads that truly can run anywhere Example: - 
cluster-a
- 
cluster-b
- 
shared
Then both workers can poll 
shared
, but keep critical jobs pinned to the cluster-specific queues. --- 11. Common gotchas - Both workers polling the default queue by accident - Missing 
queue_name
in deployment, so runs go to default queue - Worker service account can create jobs but cannot delete them - Different namespaces between worker and launched jobs - Image pull secret missing in one cluster - Cluster resource differences causing one cluster to fail scheduling --- 12. My recommendation for your case Given your earlier stuck 
Pending -> Cancelling
issue, I’d strongly recommend: - one pool - separate queues per cluster - unique worker names - explicit 
namespace
in job variables - TTL for finished jobs - good worker RBAC including delete permissions --- If you want, next I can give you either: 1. a full Helm values file tailored for Prefect Cloud, or 2. a full 
prefect.yaml
+ Docker image example for multi-cluster deployments.
k
@Marvin I have an external postgres db where the creds for that is stored on secret manager on aws. I have created a secret in k8s with the value in the secret manager, but how do i pass it to the prefect server helmchart for it to connect to it?
m
✅ thought for 135.1 seconds
Yes — for the Prefect server Helm chart, the usual pattern is: - store the full Postgres connection URL in a Kubernetes Secret - tell the Helm chart to use that existing secret - disable the built-in Postgres chart The important detail is: the secret should contain a key named 
connection-string
. Recommended approach If you already have a K8s secret populated from AWS Secrets Manager, make it look like this:
Copy code
yaml
apiVersion: v1
kind: Secret
metadata:
  name: prefect-db-connection
  namespace: prefect
type: Opaque
stringData:
  connection-string: <postgresql+asyncpg://USERNAME:PASSWORD@HOST:5432/DBNAME>
Then in your 
values.yaml
for the Prefect server chart:
Copy code
yaml
postgresql:
  enabled: false

secret:
  create: false
  name: prefect-db-connection
That tells the chart: - do not deploy internal PostgreSQL - do not create a new DB secret - use the existing secret named 
prefect-db-connection
--- If your secret currently stores username/password/host separately That will not usually be enough by itself for the Prefect server chart. The chart expects a secret containing the full connection URL under 
connection-string
. So if your AWS secret currently has JSON like:
Copy code
json
{
  "username": "prefect",
  "password": "secret",
  "host": "<http://mydb.xxxxx.us-east-1.rds.amazonaws.com|mydb.xxxxx.us-east-1.rds.amazonaws.com>",
  "port": "5432",
  "database": "prefect"
}
then you should either: 1. transform it into a Kubernetes secret with 
connection-string
, or 2. use External Secrets Operator templating to create that 
connection-string
key Example ExternalSecret template pattern:
Copy code
yaml
apiVersion: <http://external-secrets.io/v1beta1|external-secrets.io/v1beta1>
kind: ExternalSecret
metadata:
  name: prefect-db-connection
  namespace: prefect
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: prefect-db-connection
    template:
      engineVersion: v2
      data:
        connection-string: "postgresql+asyncpg://{{ .username }}:{{ .password }}@{{ .host }}:{{ .port }}/{{ .database }}"
  data:
    - secretKey: username
      remoteRef:
        key: my/prefect/postgres
        property: username
    - secretKey: password
      remoteRef:
        key: my/prefect/postgres
        property: password
    - secretKey: host
      remoteRef:
        key: my/prefect/postgres
        property: host
    - secretKey: port
      remoteRef:
        key: my/prefect/postgres
        property: port
    - secretKey: database
      remoteRef:
        key: my/prefect/postgres
        property: database
Then your Prefect Helm values remain:
Copy code
yaml
postgresql:
  enabled: false

secret:
  create: false
  name: prefect-db-connection
--- Concrete full example 
values.yaml
Copy code
yaml
postgresql:
  enabled: false

secret:
  create: false
  name: prefect-db-connection

server:
  replicaCount: 1

backgroundServices:
  runAsSeparateDeployment: false
If you are running HA / separate background services, you can still use the same DB secret. --- Important caveats 1. Secret key name matters The chart expects the secret to contain: - 
connection-string
2. Use asyncpg driver format The URL should look like:
Copy code
text
<postgresql+asyncpg://user:password@host:5432/dbname>
3. Disable built-in PostgreSQL Otherwise the chart may still try to provision its own DB:
Copy code
yaml
postgresql:
  enabled: false
4. Special characters in password If the password contains special characters like 
@
, 
:
, 
/
, 
%
, they must be URL-encoded in the connection string. 5. SSL parameters If your external Postgres requires SSL, append query params to the URL, for example:
Copy code
text
<postgresql+asyncpg://user:password@host:5432/dbname?ssl=require>
--- How to verify after deploy Check the secret:
Copy code
bash
kubectl get secret prefect-db-connection -n prefect -o jsonpath='{.data.connection-string}' | base64 -d
Check the server pod env wiring: ```bash
kubectl get deployment -n prefect kubectl describe pod -n prefect <prefect-server-pod-name>
Copy code
Then inspect logs:

```bash
kubectl logs -n prefect deploy/<prefect-server-deployment-name>
If the DB connection is wrong, Prefect server usually fails on startup/migrations with a DB error. --- Docs / repo references - Prefect Helm charts: prefect-helm - Prefect deployment docs: Deployments and workers If you want, paste your current: - Kubernetes secret shape, and - 
values.yaml
and I can rewrite it into the exact working format for the Prefect server chart.
k
@Marvin how can I set v2_enabled on the self hosted prefect dashboard
m
✅ thought for 9.1 seconds
Do you mean enable the v2 UI/dashboard experience for self-hosted Prefect Server? If so, I need to clarify one thing first because “`v2_enabled`” could refer to a few different settings depending on where you saw it: - a Helm chart/UI env var - a frontend feature flag - an old Prefect 2-era setting - or something in your own config Can you paste where you saw 
v2_enabled
? For example: - in the Helm values - in browser/network config - in a doc - or in an env var name If you want, I can also look up the exact supported setting for the current self-hosted dashboard once you share that snippet.
k
@Marvin yes, the v2 UI dashboard. I have the self hosted server. Tell me how i can enable the new UI? When I first got on it asked me if I wanted to try but I said no, how do I enable it again on the dashboard?
m
✅ thought for 104.7 seconds
Yes — if you dismissed the “Try the updated UI” prompt, the easiest way to get it back is to clear the browser flag that stores that dismissal. Fastest fix Open your Prefect dashboard in the browser, then open DevTools Console and run:
Copy code
javascript
localStorage.removeItem('prefect-v2-promo-dismissed');
window.location.reload();
That should allow the prompt to appear again. --- If you just want to switch to the new UI directly Prefect also tracks UI selection in the browser, so if the v2 UI is available on your server, clearing the promo dismissal is usually enough. If not, also try clearing site storage for your Prefect domain: - browser settings - site data / storage - clear local storage + cookies for that host --- Server-side settings for self-hosted Prefect Server There are also server settings that affect this behavior: - 
PREFECT_SERVER_UI_SHOW_PROMOTIONAL_CONTENT=true
- 
PREFECT_SERVER_UI_V2_ENABLED=true
If you’re using Helm, that would typically look like:
Copy code
yaml
server:
  config:
    PREFECT_SERVER_UI_SHOW_PROMOTIONAL_CONTENT: "true"
    PREFECT_SERVER_UI_V2_ENABLED: "true"
Then upgrade your release:
Copy code
bash
helm upgrade --install prefect-server prefect/prefect-server \
  -n prefect \
  -f values.yaml
What these do - `PREFECT_SERVER_UI_SHOW_PROMOTIONAL_CONTENT`: allows the “try the updated UI” banner/promo to show - `PREFECT_SERVER_UI_V2_ENABLED`: enables v2 as the default available UI for users without a saved preference --- What I’d try in your case 1. First, in browser console:
Copy code
javascript
localStorage.removeItem('prefect-v2-promo-dismissed');
window.location.reload();
2. If that doesn’t work, confirm your self-hosted server has:
Copy code
yaml
server:
  config:
    PREFECT_SERVER_UI_SHOW_PROMOTIONAL_CONTENT: "true"
    PREFECT_SERVER_UI_V2_ENABLED: "true"
3. Then clear site storage/cookies for your Prefect UI hostname and reload. If you want, I can help you patch your current Helm values.yaml with the exact settings in the right place.
k
@Marvin Where does the job submission stored? can i get the prefect server to submit them to an external messaging queue and then the worker pull from there?
m
✅ thought for 149.3 seconds
Short answer: flow-run work is stored in the Prefect database, and no, Prefect Server does not support replacing worker polling with an external broker like SQS/RabbitMQ/Kafka for normal work-pool job submission. Where job submission is stored For Prefect 3.x, scheduled work for workers lives in the Prefect database: - 
flow_run
- 
work_pool
- 
work_queue
Conceptually: - deployments create flow runs - flow runs are assigned to a work pool / queue - workers poll Prefect Server - Prefect Server reads from the DB and returns eligible scheduled runs So the queueing model is: - logical queue in the DB, not a standalone message broker How workers get work Workers do not consume from Redis/SQS/RabbitMQ/etc. They typically: 1. poll the Prefect API 2. API queries the DB for scheduled flow runs in the relevant work pool / queue 3. worker submits infrastructure for those runs So this is a polling architecture, not a broker-push architecture. Can Prefect Server submit jobs to an external messaging queue instead? Not as a supported pattern for work-pool job submission. There is not a supported configuration like: - “store scheduled runs in SQS” - “publish job submissions to Kafka” - “have workers pull from RabbitMQ instead of Prefect DB” That is not how work pools/workers are implemented today. Important nuance about Redis You may see Redis mentioned in self-hosted Prefect Server setups, but that is not the main worker job queue. Redis is used for other internal messaging-related purposes in some server configurations, especially HA/background-service patterns and cleanup-related workflows. It is not a drop-in external broker for work-pool dispatch. So: - Postgres/DB = authoritative store for scheduled work - workers = poll Prefect API - Redis ≠ external replacement for work submission queue What this means for your architecture If your goal is: - durable job submission - decoupled worker pickup - broker-based execution pipeline then Prefect’s standard self-hosted worker model does not let you swap in an external broker. Supported/realistic options 1. Use PostgreSQL as the durable backing store - this is the normal production approach - scale Prefect Server + workers around it 2. Run multiple workers against the same work pool/queues - horizontal scaling - still DB/API-polled 3. Use queue separation inside Prefect - work pools + work queues - not external broker queues, but usually enough for routing/priorities 4. Build a custom bridge outside supported Prefect behavior - e.g. your own service listens for events / scans runs / publishes to your broker / later calls Prefect APIs - possible architecturally, but this is custom glue, not native Prefect worker behavior For your earlier Kubernetes use case If your concern is reliability when jobs are stuck pending, an external queue won’t really solve the core Kubernetes scheduling issue. The more effective fixes are: - stable Postgres - clear work queue routing - better worker observability - robust K8s RBAC/resources - cleanup/recovery for stuck cancelling runs Useful docs - Work pools: Work pools - Deployments and workers: Deployments and workers If you want, I can sketch the actual self-hosted Prefect Server architecture for: - Postgres - Redis - API server - background services - workers and explain exactly where each component fits in the submission path.
k
@Marvin I have two eks clusters that i want to run my flows on. The idea is that user will submit to flow run request to the same url, then whichever cluster is free it should then run the flow. Is this Active-active? how can i acheive this for a self hosted prefect deployment
m
✅ thought for 35.1 seconds
Yes — what you’re describing is basically active-active execution across two EKS clusters: - users submit flow runs to one Prefect API URL - both clusters are available to take work - whichever worker gets capacity first runs the flow For self-hosted Prefect, the clean way to do this is: - one shared Prefect Server deployment (or HA server tier) - one shared PostgreSQL database - workers running in both EKS clusters - both sets of workers polling the same work pool / same shared queue That gives you active-active worker capacity. --- High-level architecture
Copy code
text
Users / CI / Apps
        |
        v
   One Prefect API URL
        |
        v
 Prefect Server (self-hosted)
        |
        v
   Shared PostgreSQL
        |
        v
  Work pool / queue state
      /         \
     /           \
EKS Cluster A   EKS Cluster B
  worker(s)       worker(s)
The server is the control plane. The workers in both clusters are the execution plane. --- How to achieve “whichever cluster is free” Use: - one Kubernetes work pool - one shared work queue (for flows that can run in either cluster) - workers in both clusters polling that queue Then: - users always hit the same Prefect API URL - flow runs land in the same work pool/queue - workers in either cluster can pick them up - the first available worker polls and claims the run That is effectively active-active at the worker layer. --- Important caveat This is not cluster-aware load balancing in the sense of: - checking current cluster CPU utilization centrally - then explicitly routing to the least-loaded cluster Instead, it is worker-poll-based competition: - both worker groups poll - eligible runs are claimed by an available worker - capacity naturally distributes based on worker availability and polling timing In practice, that often works well enough for active-active execution. --- Recommended self-hosted setup 1. One shared Prefect Server endpoint - self-hosted Prefect server accessible from both EKS clusters and your users - behind one stable URL / ingress / load balancer 2. One shared external PostgreSQL - required for production - both server and background services use the same DB 3. Optional Redis for HA/background messaging - useful in HA self-hosted server setups - but not the worker job queue 4. One work pool Example: - 
eks-active-active
5. One shared queue Example: - 
shared
6. Workers in both clusters poll the same queue Example: - worker deployment in cluster A polls 
shared
- worker deployment in cluster B polls 
shared
--- How the routing works If all workers poll: - pool: 
eks-active-active
- queue: 
shared
then either cluster can pick up the run. If cluster A is busy and cluster B has idle workers, cluster B will tend to pick up more runs. If both are free, either may get the run. --- Helm pattern for both clusters In both EKS clusters, deploy a worker pointed at the same work pool and same queue, but with different worker names. Example cluster A values:
Copy code
yaml
worker:
  enabled: true
  type: kubernetes

  config:
    workPool: eks-active-active
    name: worker-eks-a

  args:
    - prefect
    - worker
    - start
    - --pool
    - eks-active-active
    - --type
    - kubernetes
    - --name
    - worker-eks-a
    - --work-queue
    - shared
Example cluster B values:
Copy code
yaml
worker:
  enabled: true
  type: kubernetes

  config:
    workPool: eks-active-active
    name: worker-eks-b

  args:
    - prefect
    - worker
    - start
    - --pool
    - eks-active-active
    - --type
    - kubernetes
    - --name
    - worker-eks-b
    - --work-queue
    - shared
Then install the worker chart in both clusters. --- Flow deployment example All deployments that should run on either cluster go to the same pool/queue: ```yaml deployments: - name: shared-runner entrypoint: flows/my_flow.py:my_flow work_pool: name: eks-active-active
queue_name: shared job_variables: namespace: prefect image: my-registry/my-image:latest image_pull_policy: IfNotPresent
Copy code
Now any worker in either cluster can pick up the run.

---

*Critical Kubernetes caveat*

Because you are using *Kubernetes workers*, the infrastructure gets created in the cluster where the worker is running.

So:
- cluster A worker picks up the run -> K8s Job created in cluster A
- cluster B worker picks up the run -> K8s Job created in cluster B

That’s exactly what you want for active-active.

---

*How to know which cluster ran the job*

It helps to inject an env var per worker/cluster or use separate images/labels.

For example, cluster A work pool template or deployment variables could include:

```yaml
env:
  EXECUTION_CLUSTER: eks-a
And cluster B:
Copy code
yaml
env:
  EXECUTION_CLUSTER: eks-b
Then your flow can log it. If both clusters share the same queue, you won’t know in advance where it will land, so this is useful for observability. --- What about failover? This setup gives you active-active worker failover: - if cluster A workers are down, cluster B workers still poll and execute - if cluster B is down, cluster A still works But note: that only works if the Prefect control plane is also highly available. For true production active-active behavior, make these HA too: - Prefect API server replicas - background services deployment - PostgreSQL HA / managed Postgres - ingress / load balancer --- What this is not This is not: - a cross-cluster Kubernetes scheduler - a centralized least-loaded-cluster router - a broker-based execution queue It is: - a shared Prefect control plane - multiple workers in multiple clusters competing for runnable work That is the supported pattern. --- Best practice for your use case I’d suggest two queues, not just one: - 
shared
- 
cluster-a-only
- 
cluster-b-only
Why: - most flows can go to 
shared
- special flows that require a specific cluster can still be pinned when needed Then: - both clusters poll 
shared
- cluster A worker can also poll 
cluster-a-only
- cluster B worker can also poll 
cluster-b-only
This gives flexibility without losing active-active. --- Concerns from your earlier cancellation issue If you run active-active with one shared queue: - cancellation is still handled by the worker/cluster that picked up the run - observability becomes very important - give workers clear names - ensure both clusters have proper RBAC/delete permissions - monitor stuck 
Pending
/ 
Cancelling
jobs in both clusters --- What you need for self-hosted Prefect Minimum production components: - Prefect Server - external PostgreSQL - optionally Redis depending on your HA/server setup - one shared work pool - workers in both EKS clusters - one ingress/API URL reachable by users and workers --- Recommended deployment strategy Control plane cluster or shared environment - run Prefect Server in one stable place - use external Postgres - expose one API/UI URL Execution clusters - EKS A runs worker Helm release - EKS B runs worker Helm release - both use same API key / workspace / server URL - both poll the same shared queue --- Docs - Deployments and workers: Deployments and workers - Work pools: Work pools - Kubernetes worker usage: Run flows on Kubernetes If you want, I can give you a full reference architecture next with: - server Helm values - worker Helm values for both clusters - work pool / queue layout - and a 
prefect.yaml
for shared active-active routing.
k
@Marvin the problem is I am trying to avoid opening up communication between 2 clusters. so worker in cluseter 2 won't be able to read from cluster 1. However it can read from postgres. can i deploy two prefect servers?
m
✅ thought for 30.8 seconds
Yes — you can deploy two Prefect API/server instances against the same external Postgres, and this is actually the right direction if your goal is to avoid direct cluster-to-cluster communication. But there’s an important distinction: You do not want two isolated Prefect installations. You want two Prefect server instances sharing the same control-plane database. That means: - one logical Prefect control plane - shared Postgres - optionally shared Redis if you run HA/background services that need it - one server instance reachable from cluster 1 - another server instance reachable from cluster 2 - both reading/writing the same DB state This avoids cluster 2 needing to talk to cluster 1 directly. --- What this architecture looks like
Copy code
text
                 Users / clients
                      |
          +-----------+-----------+
          |                       |
          v                       v
   Prefect Server A         Prefect Server B
    reachable from            reachable from
      Cluster A                 Cluster B
          \                       /
           \                     /
            +------ Postgres ----+
                   (shared)
Then: - Worker in cluster A talks only to Server A - Worker in cluster B talks only to Server B - both server instances use the same Postgres - both see the same flow runs, work pools, queues, deployments, etc. So yes, this is a valid way to avoid direct inter-cluster communication. --- Important caveat: shared Redis / messaging may matter For modern self-hosted Prefect HA patterns, shared Postgres is required, and depending on your exact setup, shared Redis may also be needed for some background-service/messaging behavior. So the production-safe pattern is usually: - shared Postgres - shared Redis if your self-hosted HA topology uses it - multiple server/API replicas or separate server deployments - workers in both clusters pointing to their nearest/reachable server URL If you deploy two servers with the same Postgres but inconsistent supporting components/configuration, you may get odd behavior. --- What you must not do Do not do this: - Server A with its own Postgres - Server B with its own Postgres That would create two separate Prefect installations, not one shared active-active system. Workers would not share work in that model. --- Can workers in both clusters still pull the same work? Yes, if both Prefect server instances share the same Postgres state. Since work-pool/work-queue scheduling lives in the DB, both server instances can return eligible work to workers that poll them. So: - worker in cluster A asks Server A for runnable work - worker in cluster B asks Server B for runnable work - both servers query the same DB - whichever worker claims the run first gets it That gives you the “whichever cluster is free” behavior without cross-cluster worker communication. --- Is this active-active? Yes, that is effectively active-active on the control plane and worker plane, provided: - both server instances are live - both are backed by the same DB - workers in both clusters are polling - your ingress/DNS/client model supports submission to either or both API endpoints as desired --- Key design question: one URL or two URLs? You said users submit to the same URL. There are two ways to do that: Option 1: one global/load-balanced URL in front of both servers - 
<https://prefect.example.com>
- routes to Server A or Server B - both servers use same Postgres This is ideal if clients can reach both server endpoints through a common LB/DNS layer. Option 2: separate URLs per cluster/server - 
<https://prefect-a.example.com>
- 
<https://prefect-b.example.com>
Workers use their local/reachable one, but users would not have a single submission URL unless you place something in front.
If you want one submission URL without cross-cluster communication, the cleanest design is often: - public/shared LB or ingress layer - both backend servers registered behind it - no direct cluster-to-cluster traffic needed --- How to avoid cross-cluster communication This is the clean pattern: - Cluster A hosts 
prefect-server-a
- Cluster B hosts 
prefect-server-b
- both connect outbound to shared Postgres - optionally both connect outbound to shared Redis - workers in each cluster connect only to the local server endpoint - users hit a shared public URL or any reachable server URL No cluster needs to call the other cluster internally. --- Potential issues / caveats 1. Database migrations Only one migration process should manage schema upgrades at a time. 2. Background services duplication You need to be careful if both server deployments run all background services independently. Depending on your topology, this may be okay if Prefect supports coordinated multi-instance behavior through shared state, but you should avoid unmanaged duplication of migration/initialization hooks. 3. Redis/shared messaging If your topology uses Redis-backed internal messaging, it should also be shared, not per-cluster. 4. Version parity Both server deployments should run the same Prefect version. 5. Sticky debugging A worker may pick up a flow via server A while another run is queried via server B. That’s okay with shared DB, but logs/ops should assume a distributed control plane. --- Recommended architecture for your case Given your network constraint, I’d recommend: - external managed Postgres accessible from both clusters - external managed Redis accessible from both clusters if using HA/background messaging features - Prefect Server deployment in cluster A - Prefect Server deployment in cluster B - both configured identically against the same DB/Redis - one shared work pool and shared queue for “run anywhere” - workers in each cluster point only to their local Prefect server URL - optionally one public DNS name in front of both API servers --- How work claiming behaves Because claiming happens against shared DB-backed scheduling state, you don’t need worker-to-worker communication or server-to-server communication for the basic shared-queue model. The shared DB is the coordination point. That’s the key architectural insight for your use case. --- Practical answer So to your exact question: Can I deploy two Prefect servers? Yes — as long as they are two instances of the same Prefect control plane backed by the same external Postgres (and shared supporting services as needed), not two separate independent installations. That is the approach I would use to avoid opening direct communication between the clusters. --- What I would not promise without checking your exact chart/setup I’d want to verify your exact Helm topology for: - API server - background services - Redis - migrations job before telling you exactly which components to duplicate in both clusters vs run only once. That part matters. If you want, I can help you design the exact multi-cluster self-hosted topology next, including: - which Prefect Helm components should run in both clusters - which should run once only - how to wire shared Postgres/Redis - and how workers should point to the nearest server endpoint.
k
@Marvin if the pod for my flow run is pending due to failure to schedule, cancelling that workflow will simply get stuck in cancelling. Is there a way to resolve this?
m
✅ thought for 16.2 seconds
Yes — this is a known bad state with Kubernetes workers: if the job/pod never really starts, Prefect may move the flow run to Cancelling but never finish cleanup. What to do in practice 1. Delete the Kubernetes job/pod manually If the Job exists:
Copy code
bash
kubectl get jobs -n <namespace>
kubectl describe job <job-name> -n <namespace>
kubectl get pods -n <namespace> -l job-name=<job-name>
Then delete it:
Copy code
bash
kubectl delete job <job-name> -n <namespace>
If needed, force it:
Copy code
bash
kubectl delete job <job-name> -n <namespace> --grace-period=0 --force
kubectl delete pods -n <namespace> -l job-name=<job-name> --grace-period=0 --force
2. Force the Prefect flow run state to 
Cancelled
 If the infrastructure is already gone or there was never anything real to kill, force-close the Prefect state:
Copy code
python
import asyncio
from uuid import UUID
from prefect import get_client
from prefect.states import Cancelled

FLOW_RUN_ID = UUID("YOUR-FLOW-RUN-ID")

async def force_cancel():
    async with get_client() as client:
        await client.set_flow_run_state(
            FLOW_RUN_ID,
            Cancelled(message="Manually cancelled after stuck Cancelling"),
            force=True,
        )

asyncio.run(force_cancel())
This fixes the Prefect state even if cleanup logic got stuck. --- How to prevent it The real fix is to address why the pod is Pending. For Kubernetes, the usual causes are: - insufficient CPU / memory - node selector or affinity mismatch - missing tolerations - namespace quota exceeded - image pull failures - RBAC or namespace issues Check:
Copy code
bash
kubectl describe pod <pod-name> -n <namespace>
Look at Events for: - 
FailedScheduling
- 
ImagePullBackOff
- 
ErrImagePull
- 
Forbidden
- quota errors --- Important practical point If the pod never scheduled, there may be nothing running to terminate. In that case, cancelling from Prefect is mostly a metadata/state transition problem, not an actual process-kill problem. So the recovery pattern is usually: - clean up K8s objects if they exist - force the Prefect run state to 
Cancelled
--- Operational workaround A lot of teams add a cleanup job that periodically: - finds runs stuck in 
Cancelling
for > N minutes - checks if the Kubernetes job/pod still exists - if not, force-sets the state to 
Cancelled
That’s often the safest short-term mitigation. --- Bottom line Yes, there is a way to resolve it: manually remove the K8s job/pod if present, then force the Prefect flow run to 
Cancelled
. If you want, I can also give you a small automation script that: - takes a flow run ID - finds the related K8s job - deletes it - and force-cancels the Prefect run.
k
@Marvin I am hosting prefect on eks with one pod for my server connected to an external rds postgres on aws. How can i enable the clean up of the database to make sure it doesn't fill up quickly?
m
✅ thought for 110.7 seconds
Yes — for self-hosted Prefect Server backed by external Postgres, the main built-in cleanup you want is the database vacuum service. For your setup, the key thing is: - events cleanup is generally on by default - flow run cleanup is not enabled by default - you should explicitly enable DB vacuum for both events and flow runs Recommended minimum settings Set these environment variables on your Prefect server/background-services pod:
Copy code
bash
PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED=events,flow_runs
PREFECT_SERVER_SERVICES_DB_VACUUM_LOOP_SECONDS=3600
PREFECT_SERVER_SERVICES_DB_VACUUM_RETENTION_PERIOD=2592000
PREFECT_EVENTS_RETENTION_PERIOD=7d
What those mean: - 
DB_VACUUM_ENABLED=events,flow_runs
enables cleanup of: - old flow runs - orphaned logs - orphaned artifacts - old events - 
DB_VACUUM_LOOP_SECONDS=3600
run cleanup every hour - 
DB_VACUUM_RETENTION_PERIOD=2592000
retain flow runs for 30 days - 
PREFECT_EVENTS_RETENTION_PERIOD=7d
retain events for 7 days If you run a lot of flows, you may want to be more aggressive: - flow runs: 7–14 days - events: 1–3 days --- For Helm values If you are using the Prefect server Helm chart, add these under the server config/env area. A representative example:
Copy code
yaml
server:
  config:
    PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED: "events,flow_runs"
    PREFECT_SERVER_SERVICES_DB_VACUUM_LOOP_SECONDS: "3600"
    PREFECT_SERVER_SERVICES_DB_VACUUM_RETENTION_PERIOD: "2592000"
    PREFECT_EVENTS_RETENTION_PERIOD: "7d"
If your chart version structures env differently, the equivalent is to inject them as container env vars. --- Important caveat for your topology You said you have one pod for my server. That means the API server and background services may be running together in that same process/pod depending on how you started it. The cleanup service only works if server services are actually running. If you started Prefect with normal server startup, that’s usually fine. If you started with 
--no-services
, then cleanup will not run unless you have a separate background-services process. So the first thing to confirm is: - are server background services enabled? --- How to verify cleanup is active Look at server logs for database vacuum activity:
Copy code
bash
kubectl logs -n <namespace> deployment/<prefect-server-deployment>
You want to see messages related to: - database vacuum - deleting old flow runs - deleting old events Also check current settings inside the container if needed. --- What gets cleaned up With DB vacuum enabled, Prefect cleans up things like: - old terminal flow runs - orphaned logs - orphaned artifacts - stale event data This is the main built-in answer to “how do I keep the DB from filling up too quickly?” --- What usually grows fastest In many self-hosted setups, the fastest-growing data is: 1. events 2. logs 3. flow/task run history So the biggest wins are usually: - lower 
PREFECT_EVENTS_RETENTION_PERIOD
- enable 
flow_runs
vacuum - reduce unnecessary logging volume in flows --- Recommended retention starting point For moderate usage:
Copy code
bash
PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED=events,flow_runs
PREFECT_SERVER_SERVICES_DB_VACUUM_LOOP_SECONDS=3600
PREFECT_SERVER_SERVICES_DB_VACUUM_RETENTION_PERIOD=1209600
PREFECT_EVENTS_RETENTION_PERIOD=3d
That means: - clean hourly - keep flow runs for 14 days - keep events for 3 days For high-volume environments, this is often much better than 30–90 day retention. --- Postgres-side recommendation Even with Prefect cleanup enabled, Postgres still needs its own vacuum/autovacuum to reclaim space efficiently. So also make sure on RDS: - autovacuum is enabled - monitor table growth and dead tuples - alert on storage growth Prefect deletes rows, but Postgres storage reclamation behavior still matters. --- If you want more aggressive cleanup than built-in retention
You can also run your own scheduled maintenance flow or DBA job, but I’d start with the built-in DB vacuum first. --- Good next step for you Because you’re on EKS + RDS with a single server pod, I’d recommend: 1. enable:
Copy code
yaml
server:
  config:
    PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED: "events,flow_runs"
    PREFECT_SERVER_SERVICES_DB_VACUUM_LOOP_SECONDS: "3600"
    PREFECT_SERVER_SERVICES_DB_VACUUM_RETENTION_PERIOD: "1209600"
    PREFECT_EVENTS_RETENTION_PERIOD: "3d"
2. redeploy Helm 3. verify server logs show vacuum activity 4. monitor RDS table growth for a few days --- References - Prefect self-hosted/server docs: Self-host Prefect - Helm charts: prefect-helm If you want, I can help you patch these exactly into your current Helm values file based on how you installed Prefect server.
k
@Marvin i don't see database logs in my prefect server? I deployed it using helm - how do i enable the background service?
m
✅ thought for 15.0 seconds
If you installed Prefect Server via Helm and you don’t see database-vacuum/background-service logs in the main server pod, the most likely reason is: - your chart is running API server only, or - background services are split out and not enabled/configured the way you expect For Helm-based self-hosted Prefect, the clean pattern is usually: - API server deployment - background services deployment (separate) - shared Postgres - optionally Redis for HA messaging features So to get DB cleanup running reliably, you generally want to enable the separate background services deployment. What to enable in Helm Add this to your `values.yaml`:
Copy code
yaml
backgroundServices:
  runAsSeparateDeployment: true
  replicaCount: 1

server:
  config:
    PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED: "events,flow_runs"
    PREFECT_SERVER_SERVICES_DB_VACUUM_LOOP_SECONDS: "3600"
    PREFECT_SERVER_SERVICES_DB_VACUUM_RETENTION_PERIOD: "1209600"
    PREFECT_EVENTS_RETENTION_PERIOD: "3d"
Then upgrade the release:
Copy code
bash
helm upgrade --install prefect-server prefect/prefect-server \
  -n prefect \
  -f values.yaml
What this does - 
backgroundServices.runAsSeparateDeployment: true
creates a dedicated background-services pod - that pod runs the internal services like: - database vacuum - cancellation cleanup - lease cleanup / service loops - the DB cleanup env vars tell it to retain less data --- How to verify it worked After upgrade:
Copy code
bash
kubectl get pods -n prefect
You should see something like: - server pod - background services pod Then inspect logs for the background deployment/pod, not just the server pod:
Copy code
bash
kubectl logs -n prefect deployment/<background-services-deployment-name>
or:
Copy code
bash
kubectl get pods -n prefect
kubectl logs -n prefect <background-services-pod-name>
That’s where you’d expect to see vacuum-related activity. --- Why you may not see it in the server pod In Helm deployments, background services are often separated so the API process doesn’t also own all service loops. That’s normal and usually preferable. --- One important caveat If you enable separate background services in a more HA-style setup, some Prefect topologies also expect Redis/shared messaging support for full coordination. If you’re still on a simple single-server deployment, you may still be okay, but if you scale beyond that, I’d strongly recommend reviewing Redis/shared service coordination too. --- What I’d do in your case Since you currently have: - EKS - Helm-installed Prefect server - external RDS - concern about DB growth I would: 1. enable 
backgroundServices.runAsSeparateDeployment: true
2. set the DB vacuum retention env vars 3. redeploy 4. inspect logs of the background services pod --- If you want, send me your current Helm 
values.yaml
 and I can mark up the exact section to add so you don’t have to guess the chart structure.
k
@Marvin do i need to have an external redis? what is it for?
m
✅ thought for 20.7 seconds
Short answer: not always. For your setup, external Redis is not strictly required unless you are running Prefect in a more distributed / HA-style topology. What Redis is for in self-hosted Prefect In Prefect 3.x self-hosted deployments, Redis is not the main job queue for workers. Instead, it is used for certain internal coordination and messaging functions in more advanced deployments, such as: - server/background-service coordination - messaging/caching for distributed components - cleanup queue / retry-style internal coordination - some event ordering / lease-storage patterns in HA setups So think of Redis as: - internal coordination layer - not the worker work-pool backend --- For your current setup You said you have: - EKS - one Prefect server pod - external RDS Postgres In that case, you likely do not need external Redis yet if: - you run a simple single-server setup - you are not scaling API/background services across multiple replicas/clusters - you are okay with a simpler architecture You can often run Prefect Server with just: - Prefect server - Postgres and no Redis. --- When Redis becomes more important You should consider external Redis if you want: - multiple Prefect server/background-service instances - more HA/distributed self-hosted architecture - better coordination across multiple server pods/clusters - features/configurations that depend on shared messaging/lease storage Given your earlier multi-cluster questions, Redis becomes much more relevant if you move toward: - multiple Prefect server instances - separate background-services deployments - active-active style control plane In that case, shared Redis can help provide coordination that a single in-cluster Redis per cluster would not. --- Do you need it just for database cleanup? No, not necessarily. If you just want: - background services - DB vacuum / retention cleanup - one self-hosted server environment you may be able to do that without external Redis. --- Rule of thumb - Single server pod + Postgres: Redis usually not required - Separate background services, multiple API pods, multi-cluster control plane: Redis becomes much more advisable / sometimes necessary depending on configuration --- Why this matters for your architecture Since you were exploring: - two EKS clusters - maybe two Prefect servers - shared Postgres - no direct cluster-to-cluster communication then I would say: - shared Postgres is mandatory - shared external Redis is strongly recommended if you want a robust multi-instance self-hosted control plane Without Redis, a simple deployment may still work, but the more distributed you get, the more likely you are to hit coordination edge cases. --- So for your exact question Do I need an external Redis? - No for a simple single-server self-hosted deployment - Probably yes / strongly recommended for multi-server or multi-cluster HA-style Prefect Server What is it for? - internal messaging / coordination for distributed server components - not for storing flow-run work for workers --- If you want, I can next give you a decision table for: 1. single EKS cluster + one server pod 2. single cluster + separate background services 3. two clusters + two Prefect server instances and say exactly whether Redis is required / recommended / optional for each.
k
@Marvin how do i know if i need multiple server pods or not?
m
✅ thought for 21.2 seconds
Good question. The short answer is: You need multiple server pods if you need high availability or more API capacity. If you just want a working self-hosted Prefect for modest usage, one server pod is often enough. How to decide Ask these questions: ### 1. Do you need high availability? If one Prefect server pod dies, what happens? - If it’s okay for the UI/API to be briefly unavailable while Kubernetes restarts the pod, one pod may be fine - If you need the API/UI to remain available during node/pod failure or rolling deploys, you want multiple server pods Use multiple pods when: - Prefect is business-critical - users/automations need continuous API availability - you want zero/low-downtime upgrades --- ### 2. How much API traffic do you have? A single server pod is usually enough for small to medium setups. You may want multiple pods if you have: - many users on the UI - many workers polling frequently - many flow/task state updates - many automations/events/logs - slow responses or API saturation Symptoms that suggest scaling server pods: - API requests are slow - workers time out or struggle polling - UI feels sluggish - server CPU/memory is consistently high - pod restarts under load --- ### 3. How many workers / flow runs are you running? There’s no universal cutoff, but rough intuition: - small: a few workers, tens/hundreds of runs per day -> one pod usually fine - medium: many workers, frequent scheduling/state updates -> maybe still one, but monitor - large / multi-team / always-on platform: multiple pods usually makes sense --- ### 4. Are you doing multi-cluster active-active control plane? If you want: - two EKS clusters - server reachable from both - no single-cluster control-plane dependency then multiple server instances/pods becomes much more relevant. If you keep only one server pod in one cluster, then that cluster is still your control-plane single point of failure, even if flows can run in both clusters. --- ### 5. Do you separate API and background services? If you begin running: - API server separately - background services separately then you are already moving toward a more production-grade topology, where scaling API pods independently can make sense. --- When one pod is enough One server pod is usually enough if: - this is internal / low criticality - a short outage during restart is acceptable - you have a modest number of users and workers - you are primarily trying to keep the setup simple - you’re not trying to build HA across clusters This is the “simple and reasonable” setup. --- When multiple server pods are worth it Use multiple server pods if: - you need HA for the API/UI - you expect rolling upgrades without downtime - you have many workers polling - you have multiple teams using Prefect - you want more robust self-hosted production behavior - you’re building a shared platform service --- Important distinction Even with multiple server pods: - Postgres remains the main source of truth - adding more pods does not solve DB performance issues by itself - if the DB is the bottleneck, scaling server pods won’t help much So before scaling server pods, monitor: - server CPU/memory - response latency - DB CPU/connections/storage/slow queries - worker polling behavior --- Practical recommendation for your case Based on what you’ve described so far: - self-hosted on EKS - one server pod - external RDS - interest in two clusters / active-active execution - desire to avoid unnecessary complexity I’d say: ### Stay with one server pod if: - you’re still validating the platform - downtime during restart is acceptable - API load is modest - you do not yet need HA control plane ### Move to multiple server pods if: - Prefect is becoming shared production infrastructure - you want the API to stay available during pod/node failure - you want multi-cluster control plane access - you’re seeing real API pressure --- My rule of thumb
- Dev / small prod: 1 server pod - serious production shared service: 2+ API pods, separate background services, external Postgres, likely Redis - multi-cluster HA control plane: multiple server instances/pods plus shared backing services --- What I would do if I were you Given your current stage, I would likely: 1. keep one server pod for now 2. enable background services / DB cleanup properly 3. monitor load and failure modes 4. only add multiple server pods when you actually need: - HA - scale - multi-cluster control plane That avoids premature complexity. --- Simple decision test If the answer to either of these is yes, start planning multiple server pods: - “Would a 2–5 minute Prefect API outage be unacceptable?” - “Am I building this as a shared production platform for multiple teams/clusters?” If both are no, one pod is probably still fine. If you want, I can give you a recommended self-hosted architecture progression: - phase 1: one pod - phase 2: one pod + background services - phase 3: HA multi-pod/multi-cluster so you can decide when to graduate between them.
k
@Marvin is there a way to have all the logs inside k8s to show u pon the prefect UI? including third party packages
m
✅ thought for 185.5 seconds
Yes — mostly, but not automatically for all possible logs. For flows running on Kubernetes workers, logs show up in the Prefect UI when they are captured by Prefect’s Python logging system and sent to the API. So the practical answer is: - Prefect flow/task logs → yes - 
print()
output → yes, if enabled - third-party Python package logs → yes, if you explicitly configure those loggers - arbitrary container stdout/stderr from non-Python processes → not always automatically - crash logs from failed K8s pods → often yes, via the Kubernetes observer feature --- What to enable ### 1. Enable log sending to the Prefect API Usually this is already on, but you want:
Copy code
bash
PREFECT_LOGGING_TO_API_ENABLED=true
### 2. Capture 
print()
statements At the flow level:
Copy code
python
from prefect import flow

@flow(log_prints=True)
def my_flow():
    print("hello from print")
Or globally with env:
Copy code
bash
PREFECT_LOGGING_LOG_PRINTS=true
--- ### 3. Capture third-party Python logger output This is the key piece for packages like: - 
requests
- 
urllib3
- 
sqlalchemy
- etc. Set:
Copy code
bash
PREFECT_LOGGING_EXTRA_LOGGERS=requests,urllib3,sqlalchemy
That tells Prefect to attach its API log handler to those named loggers. --- Example Kubernetes job env / work-pool job variables If you configure this in your deployment/job variables:
Copy code
yaml
job_variables:
  env:
    PREFECT_LOGGING_TO_API_ENABLED: "true"
    PREFECT_LOGGING_LOG_PRINTS: "true"
    PREFECT_LOGGING_LEVEL: "INFO"
    PREFECT_LOGGING_EXTRA_LOGGERS: "requests,urllib3,sqlalchemy"
Then those logs should appear in the Prefect UI. --- Example Python flow
Copy code
python
import logging
import requests
from prefect import flow, get_run_logger

@flow(log_prints=True)
def my_flow():
    logger = get_run_logger()
    <http://logger.info|logger.info>("Starting flow")

    print("This print will appear in Prefect UI")

    third_party_logger = logging.getLogger("requests")
    third_party_logger.setLevel(<http://logging.INFO|logging.INFO>)

    requests.get("<https://example.com>")

    <http://logger.info|logger.info>("Done")
--- Important limitation Prefect does not magically ingest every log emitted inside the pod. It primarily captures logs that go through: - Prefect loggers - configured Python loggers - captured 
print()
- some crash-log forwarding from the Kubernetes observer If a process inside the container writes directly to stdout/stderr and is not integrated with Python logging / Prefect logging, those logs may remain only in Kubernetes logs unless you explicitly bridge them. --- For crashed runs / startup failures Kubernetes workers also support forwarding logs from crashed pods, which is especially useful when the flow never starts cleanly. This helps with cases like: - import errors - bad image/entrypoint - missing dependencies - crash on startup Relevant settings include enabling the Kubernetes observer crash-log forwarding. In practice, you want this enabled if you're debugging failed jobs. --- Best practice for “show everything possible in UI” Use all three: 1. 
@flow(log_prints=True)
2. 
PREFECT_LOGGING_EXTRA_LOGGERS=...
3. set log levels appropriately for those libraries For example:
Copy code
bash
PREFECT_LOGGING_TO_API_ENABLED=true
PREFECT_LOGGING_LOG_PRINTS=true
PREFECT_LOGGING_LEVEL=INFO
PREFECT_LOGGING_EXTRA_LOGGERS=requests,urllib3,sqlalchemy,boto3,botocore
And in code, if needed:
Copy code
python
import logging
logging.getLogger("urllib3").setLevel(<http://logging.INFO|logging.INFO>)
logging.getLogger("sqlalchemy").setLevel(<http://logging.INFO|logging.INFO>)
--- If you want all pod logs regardless of source If your goal is truly: - everything written in the pod - including shell commands, subprocess output, non-Python processes then Prefect UI is not really a full log aggregation platform. In that case, the usual pattern is: - send pod/container logs to CloudWatch / Datadog / ELK / Loki - use Prefect UI for orchestration logs - use your cluster log platform for full raw container logs
That’s the most robust setup. --- Recommended split - Prefect UI: flow/task/application logs relevant to orchestration - Kubernetes log stack: full container/system/subprocess/raw stdout logs --- Docs - Prefect deployments/workers: Deployments and workers - Kubernetes worker docs: Run flows on Kubernetes If you want, I can give you a concrete Prefect deployment/job_variables example for EKS that captures: - 
print()
- 
requests
- 
sqlalchemy
- 
boto3
- and sets sensible logging levels.
13 Views
Open in Slack
PreviousNext
Powered by