Based on the research and GitHub issues, here are the most common reasons why your tasks might be stuck in pending state when using Kubernetes workers in Prefect 2.x: 1. Resource Constraints: - Insufficient CPU or memory resources in your Kubernetes cluster - Incorrect resource requests and limits in your job manifests - Pods might be unable to schedule due to resource quotas 2. Kubernetes Configuration Issues: - Incorrect namespace permissions - Missing or incorrectly configured service accounts - Issues with image pull secrets or policies - Network connectivity problems preventing image pulls 3. Worker-Related Issues: - Worker crashes due to API server overload (as seen in GitHub issue #16977) - Worker restarts while tasks are being scheduled can leave tasks in a pending state - Multiple workers trying to schedule the same tasks simultaneously To troubleshoot this, I recommend: 1. Check Kubernetes Events:

bash
   kubectl get events --sort-by='.lastTimestamp' -n your-namespace

This will show if there are scheduling issues, resource constraints, or permission problems. 2. Verify Pod Status:

bash
   kubectl get pods -n your-namespace
   kubectl describe pod <pod-name> -n your-namespace

Look for events or warnings that might indicate why pods aren't starting. 3. Check Worker Logs:

bash
   kubectl logs -n your-namespace <worker-pod-name>

Look for any error messages or warnings about task scheduling. 4. Verify Resources: - Check your cluster's available resources - Verify the resource requests and limits in your work pool configuration - Ensure you haven't hit any namespace quotas 5. Check Prefect Worker Health:

bash
   prefect worker ls

Verify that your workers are healthy and properly connected. Would you be able to share: 1. Are you seeing any specific error messages in your Kubernetes events or worker logs? 2. How many workers do you have running? 3. Are you using Prefect Cloud or a self-hosted server? This information would help me provide more specific guidance for your situation. Also, since you're using Prefect 2.x, I should mention that Prefect 3.x has significant improvements in worker reliability and task scheduling. If possible, consider upgrading to take advantage of these improvements.

Based on this new context, I can provide more targeted troubleshooting advice. If tasks are meant to run in the same pod as the flow, there are a few specific things to check: 1. Task Runner Configuration: - Make sure you're using the correct task runner for in-pod execution - The default

ConcurrentTaskRunner

or

SequentialTaskRunner

should be used for same-pod execution - Check if you're accidentally using a

KubernetesTaskRunner

which would try to spawn new pods 2. Flow Configuration: Could you share your flow code? It would be helpful to see how the tasks are defined and submitted. Here's what to check in your code:

python
   from prefect import flow, task
   
   @task
   def my_task():
       # task code
       pass
   
   @flow
   def my_flow():
       # This will run in the same pod
       result = my_task()
       
       # This might cause issues if not configured correctly
       result = my_task.submit()

3. Common Issues to Check: - Are you using

.submit()

on your tasks? This can sometimes cause issues if not configured correctly - Are you using any specific task runners or flow runners? - Are there any concurrent execution settings that might be affecting task execution? To help diagnose this further, could you please share: 1. A simplified version of your flow code showing how tasks are defined and called 2. Are you seeing any error messages in your flow logs? 3. How many tasks are you trying to run simultaneously? 4. Are you using any specific task runners or flow configuration? This information would help me provide more specific guidance for your situation. Also, if you're using task submission with

.submit()

, you might want to check if there are any concurrency limits set that could be preventing task execution. In Prefect 2.x, these settings can sometimes cause tasks to remain in a pending state if not configured correctly.

I’m missing the context—what question are you referring to? If this is about a Prefect issue, could you share: - What you’re trying to do (briefly) - Prefect version (2.x or 3.x) - How you’re running it (local, Cloud/Server, worker type, Docker/K8s) - The exact error/logs (last 20–30 lines help) - Minimal code snippet or CLI command you ran With that, I can give you a concrete answer.

Short answer: there isn’t a single “health” command in Prefect 2.x, but there are a few quick ways to verify you’re pointed at — and can talk to — the right API. Quick checks: - Verify your config (URL/key/profile) - Run:

prefect config view

Confirm

PREFECT_API_URL

(and

PREFECT_API_KEY

for Cloud) match what you expect. - Hit the API via a CLI command that queries the server - For any environment (OSS or Cloud):

prefect deployment ls

If you’re connected, you’ll get a list (or “no deployments” if empty). If you’re not, you’ll see a connection/auth error. - If you’re on Prefect Cloud specifically:

prefect cloud workspace ls

This confirms your API key/URL/workspace are valid. - Ping the health endpoint (useful for self-hosted/OSS) - If your

PREFECT_API_URL

includes

/api

(typical), try:

curl "$PREFECT_API_URL/health"

You should get a 200 response with a small JSON payload. - Minimal Python client check (async in 2.x)

from prefect import get_client
  import asyncio

  async def check():
      async with get_client() as client:
          # a lightweight call that must hit the API
          runs = await client.read_flow_runs(limit=1)
          print("OK, got response from API.", len(runs))

  asyncio.run(check())

Helpful docs: - Settings and config (prefect config view) - Connecting to Prefect Cloud - Using the Prefect client Follow-up: Are you on Prefect Cloud or a self-hosted Prefect 2.x server? I can tailor the exact check and expected outputs.

Got it — a few likely culprits here. First, a quick note: you wrote `PREFECT_SERVER_APU_HOST`; if that’s in your CI variables, it’s a typo. The setting is

PREFECT_SERVER_API_HOST

. That said, for clients calling the API in Prefect 2.x, only

PREFECT_API_URL

really matters. Why 200 locally but “0” in CI? - In Prefect 2.x, a “status 0” typically means a network-level error (DNS, TLS/certs, proxy, connection refused), not an HTTP response from the server. - Common causes in CI: - Missing CA certificates in the container (very common with Alpine or minimal base images) - Proxy settings causing the request to fail (missing

HTTPS_PROXY

or missing

NO_PROXY

for an internal host) -

PREFECT_API_URL

pointing at

localhost

or a private host not reachable from the CI runner - Different job/image in CI for “deploy” vs “read” steps; the deploy job has network/auth, the other doesn’t Fast checks to run inside the same CI job that fails: - Print config to verify the values the process actually sees:

prefect config view

- Test the health endpoint:

echo "$PREFECT_API_URL"
  curl -v "$PREFECT_API_URL/health"

- If adding

-k

(insecure) makes it work, it’s a CA cert issue in the container. - Minimal Python probe that uses the Prefect client:

python - <<'PY'
  import os, asyncio, traceback
  from prefect import get_client
  print("API URL:", os.getenv("PREFECT_API_URL"))
  print("API KEY set:", bool(os.getenv("PREFECT_API_KEY")))
  async def main():
      async with get_client() as client:
          try:
              deps = await client.read_deployments(limit=1)
              print("OK: got deployments:", len(deps))
          except Exception as e:
              traceback.print_exc()
  asyncio.run(main())
  PY

If it’s a cert/proxy issue: - Install CA certs in the job image: - Debian/Ubuntu:

apt-get update && apt-get install -y ca-certificates

- Alpine:

apk add --no-cache ca-certificates

- If you’re behind a proxy, set

HTTPS_PROXY

and, if your API is internal, add the hostname to

NO_PROXY

so it bypasses the proxy. If it’s a reachability issue: - Ensure

PREFECT_API_URL

is not

<http://127.0.0.1:4200/api>

unless the Prefect Server is running in the same CI network namespace and port is exposed. CI “localhost” won’t point to your Mac. - For Prefect Cloud, make sure

PREFECT_API_KEY

is present in the failing job and not restricted by GitLab’s “protected” variable settings. Next steps - Can you share: - Whether this is Prefect Cloud or self-hosted Server - The base Docker image for the failing CI job - The exact

PREFECT_API_URL

pattern (mask secrets; just confirm http vs https and host shape) - The traceback from the Python probe above - If it is Alpine or a minimal image and you’re using

<https://api.prefect.cloud/>...

, installing

ca-certificates

usually resolves the “status 0” symptom. Helpful refs: - Prefect 2 settings and config - Using the Prefect client in 2.x

Ah—got it. So the call succeeds but returns an empty list in CI. Most common reasons for “0 deployments” in Prefect 2.x when it works locally: - Different workspace: CI is pointed at a different account/workspace than your local env. - Filters: Your code may be applying a filter (e.g., by work pool name) that doesn’t match anything in CI. - Different profile/config at runtime: The job that deploys and the job that lists may be using different Prefect profiles or env vars. Quick checks (run these inside the same CI job that returns 0): - Confirm config and workspace

prefect version
  prefect config view
  echo "$PREFECT_API_URL"

If you’re on Cloud also:

prefect cloud workspace show

Ensure the API URL has the expected account_id/workspace_id. - Minimal Python probe (no filters)

python - <<'PY'
  import asyncio
  from prefect import get_client
  async def main():
      async with get_client() as client:
          print("Client API URL:", getattr(client, "api_url", None))
          deps = await client.read_deployments(limit=5)
          print("Found deployments:", len(deps))
          for d in deps:
              print(getattr(d, "id", None), getattr(d, "name", None), getattr(d, "flow_name", None))
  asyncio.run(main())
  PY

If this still returns 0: - You’re likely in a different workspace. Compare the

PREFECT_API_URL

from CI vs local. In Cloud it should look like:

<https://api.prefect.cloud/api/accounts/<ACCOUNT_ID>/workspaces/<WORKSPACE_ID>>

- Or your code applies filters. For example, a work pool filter that doesn’t exist in CI will yield 0:

from prefect.client.schemas.filters import WorkPoolFilter, WorkPoolFilterName
  # If "my-pool" doesn’t exist in CI, this returns 0
  # await client.read_deployments(work_pool_filter=WorkPoolFilter(name=WorkPoolFilterName(any_=["my-pool"])))

Notes: - You mentioned

PREFECT_SERVER_APU_HOST

— that’s a typo. It’s

PREFECT_SERVER_API_HOST

, but for a client call you generally only need

PREFECT_API_URL

(and

PREFECT_API_KEY

for Cloud). I’d remove

PREFECT_SERVER_*

vars from the client environment to avoid confusion. - If the deploy step “worked” in CI, make sure your list step runs in the same job image and with the same env/profile; different jobs often miss env vars or use different profiles. If you can share: - Cloud vs self-hosted - The

PREFECT_API_URL

pattern (http/https + account/workspace shape) - The exact read code (especially any filters) -

prefect version

in both environments …I can pinpoint the mismatch quickly. Refs: - Prefect 2 settings and config - Connecting to Prefect Cloud - Using the Prefect client