<@ULVA73B9P> I am testing a new infrastructure con...
# ask-marvin
m
@Marvin I am testing a new infrastructure configuration for eks with karpenter. I was able to deploy my workflow but when I try to run it, it stays stuck in late and dashboard says "work queue does not have any actively polling workers ready to execute work". What do you suggest I do?
m
thought for 374.4 seconds
That message means Prefect doesn’t see any worker actively polling the work queue your deployment is targeting. On EKS with Karpenter this is usually either “no worker is running” or “the worker can’t reach Prefect (API/auth mismatch)”. Quickest path to green: 1) Confirm a worker is online for the right work pool - In the UI: Work Pools → your pool → Workers tab. You should see at least one worker ONLINE. - Or CLI:
Copy code
prefect worker ls
prefect work-pool ls
2) If no worker is running, start one - Temporarily (local test):
Copy code
export PREFECT_API_KEY="pnu_..."            # Cloud only
prefect worker start --pool "<your-pool>"
- Recommended for EKS: deploy the Kubernetes worker via Helm (stays online even when Karpenter scales nodes): values.yaml:
Copy code
worker:
  apiConfig: cloud
  cloudApiConfig:
    accountId: "<your-account-id>"
    workspaceId: "<your-workspace-id>"
    apiKeySecret:
      name: prefect-api-key
      key: key
  config:
    workPool: "<your-work-pool-name>"
    type: kubernetes
    # optional: jobNamespace: "prefect"  # if you want jobs in a separate ns

serviceAccount:
  create: true
Install:
Copy code
kubectl create namespace prefect
kubectl create secret generic prefect-api-key -n prefect --from-literal=key=pnu_xxx
helm repo add prefect <https://prefecthq.github.io/prefect-helm>
helm repo update
helm install prefect-worker prefect/prefect-worker -n prefect -f values.yaml
Then:
Copy code
kubectl get pods -n prefect
kubectl logs -f deploy/prefect-worker -n prefect
3) Verify your deployment points to the same pool (and queue)
Copy code
prefect deployment ls -v
- The deployment’s work_pool_name must match the pool your worker is polling. - If you created a custom work queue, either: - Start the worker without restricting queues (polls all queues in the pool), or - Start it with the specific queue name. 4) Check queue/deployment status and limits - UI: Work Pools → your queue shows READY, not Paused; Concurrency limit not 0. - UI: Deployments → your deployment is active (not paused). - If needed, resume in the UI and/or increase concurrency. 5) Confirm API connectivity from the worker - In worker pod logs you should see it “Polling work pool …”. - If not, check env: - Cloud: API key present via the secret; account/workspace IDs correct. - Self-hosted: PREFECT_API_URL points to your server and is reachable from the cluster. 6) Karpenter considerations - Your worker must be “always on” for the queue to stay READY. If Karpenter is scaling the worker away, the queue will show NOT_READY and runs will go Late. Pin the worker to a small, persistent node class or add a PDB/min replicas so it isn’t evicted during scale down. If you can, share a few details and I’ll pinpoint the fix: - Are you on Prefect Cloud or self-hosted? - The work pool name your deployment uses - How you’re running the worker (Helm? local? container?) - Output of:
Copy code
prefect deployment ls -v
prefect worker ls
kubectl get pods -n prefect
kubectl logs -n prefect deploy/prefect-worker | tail -n 50
Helpful docs: - Workers concept: Workers - Kubernetes how-to: Run flows on Kubernetes - Manage work pools/queues: Manage work pools - Prefect Helm chart (worker): prefect-worker Helm chart