Short answer: that log is emitted before the worker actually talks to Kubernetes. If you never see anything after “Worker KubernetesWorker submitting flow run”, it almost always means the worker failed during Kubernetes client setup or job creation and either crashed/restarted or couldn’t log the error. The most common root cause is missing RBAC permissions for the worker’s ServiceAccount to create Jobs (and sometimes Secrets) in the target namespace. What normally happens after that line - “Worker … submitting flow run …” - “Creating Kubernetes job…” - “Completed submission of flow run …” If you don’t see “Creating Kubernetes job…” or “Completed submission…”, the failure occurred before or during job creation. Common root causes - RBAC: ServiceAccount can’t create jobs/pods/secrets in the namespace. - Symptoms: No Job shows up. Often the flow run state flips to Crashed with a message like “jobs.batch is forbidden…”. Sometimes the worker pod restarts and you miss the error. - Kubernetes API unreachable from the worker (network policy, DNS, cluster config). - Symptoms: Long silence, then worker crash/retry. No Job created. - Namespace issues: namespace doesn’t exist or differs from the worker’s target namespace. - Secret creation failures (if the worker is configured to create a secret for PREFECT_API_KEY). - Symptoms: 403 on secrets; no Job created. - Worker image missing Kubernetes integration (kubernetes-asyncio not installed). - Install the extra: pip install "prefect[kubernetes]" - Worker pod is OOMKilled or restarts mid‑submission. - Symptoms: abrupt stop in logs, new container start. Quick triage checklist 1) Check the worker pod logs around that time for an error after “submitting flow run”. Look for “Failed to submit flow run to infrastructure”. 2) Verify a Job actually got created: - kubectl get jobs -n <namespace> 3) Verify RBAC for the ServiceAccount the worker uses: - kubectl auth can-i create jobs --as=systemserviceaccount<ns>:<sa> -n <namespace> - If you need a quick (broad) fix for testing:

kubectl create clusterrolebinding prefect-worker-edit \
       --clusterrole=edit \
       --serviceaccount=<ns>:<sa>

4) Check Events in the namespace for API or quota errors: - kubectl get events -n <namespace> --sort-by=.lastTimestamp 5) Confirm the namespace and SA configured in your work pool match what’s in the cluster. 6) If you’re using secret creation for the API key, try disabling it temporarily to rule it out. 7) Ensure the worker image includes the Kubernetes integration: - pip install "prefect[kubernetes]" If you can share: - Whether you see “Creating Kubernetes job…” or “Failed to submit flow run …” after the “submitting” line - The flow run state (Pending/Running/Crashed) and any state message - The worker’s namespace and ServiceAccount name - Output of kubectl get jobs -n <ns> and kubectl auth can-i create jobs --as=systemserviceaccount<ns>:<sa> -n <ns> …I can pinpoint the root cause quickly. Helpful references - Kubernetes RBAC overview: RBAC docs - Prefect docs home: Prefect 3.x docs (see the Kubernetes worker integration section)

Thanks for the detail — if the only line you ever see is:

Worker KubernetesWorker submitting flow run

and nothing after, that means the worker picked up the run but never got far enough to log “Creating Kubernetes job…” or to emit an error like “Failed to submit flow run…”. In practice, this points to one of these root causes: Most common root causes - The worker pod restarted mid‑submission (OOMKilled/crash) so you miss subsequent logs. - RBAC prevents creating Jobs/Secrets and the worker died/restarted before flushing the error. - Kubernetes API/network is unreachable from the worker (network policy/DNS), causing a hard hang or crash during client setup. - Secret creation for the API key (if enabled) failed early and the worker died. - The worker image is missing the Kubernetes integration and crashes when initializing the client. Targeted checks 1) Check the flow run state/message in the UI: - If it shows Crashed with “Flow run could not be submitted…”, share the message — it usually includes the Kubernetes error (e.g., jobs.batch is forbidden). - If it remains Pending or disappears without a crash message, the worker likely restarted. 2) Check if the worker pod restarted: - Describe the pod and look for OOMKilled or restarts.

kubectl describe pod <worker-pod> -n <ns>
     kubectl get pod <worker-pod> -n <ns> -o jsonpath='{.status.containerStatuses[*].restartCount}'

- If restarts > 0 or OOMKilled shows up, increase resources or inspect container logs prior to restart. 3) Verify RBAC for the worker ServiceAccount:

kubectl auth can-i create jobs --as=system:serviceaccount:<ns>:<sa> -n <target-namespace>
   kubectl auth can-i create secrets --as=system:serviceaccount:<ns>:<sa> -n <target-namespace>

If these return “no”, grant permissions (for a quick test):

kubectl create clusterrolebinding prefect-worker-edit \
     --clusterrole=edit \
     --serviceaccount=<ns>:<sa>

4) Confirm namespace and work pool settings match: - Ensure the namespace in your work pool/job template exists and is the one your SA has rights to. 5) Network reachability from worker: - Check for NetworkPolicies that could block egress to the API server. - From another pod in the same namespace/with same SA, verify basic API calls (or review Events).

kubectl get events -n <namespace> --sort-by=.lastTimestamp

6) Enable debug logs on the worker to capture early errors: - Set env:

PREFECT_LOGGING_LEVEL=DEBUG

on the worker deployment and re-test. 7) Ensure the Kubernetes integration is present in the worker image: - The image should include the extra:

pip install "prefect[kubernetes]"

. If you can share: - The flow run’s current state and any state message - Worker SA and namespace - Worker pod restart count and any OOMKilled indication - Result of the two

kubectl auth can-i

checks …I can help pinpoint the exact root cause and next fix quickly. Prefect docs home (for reference): Prefect 3.x docs

Short answer: yes — if no worker is online/connected or the worker can’t enforce cancellation, a flow run can sit in “Cancelling” indefinitely. But there are a couple of nuances worth checking. What should happen - If the run was still Scheduled/Pending/Submitting and had no infrastructure_pid, Prefect 3.x normally bypasses Cancelling and moves straight to Cancelled. - If the run is Running (or the server thinks it is), the state becomes Cancelling and a worker must kill the infra (for Kubernetes: delete the Job/Pods). Only then will the state move to Cancelled. Why you might be stuck in Cancelling - No active worker observing cancellations (worker offline, crashed, or disconnected from Prefect). - Worker has no permission to delete the Kubernetes Job/Pods (RBAC delete denied). - Missing or bad infrastructure_pid (e.g., worker failed to set it; in that case the run often cannot be cancelled by the worker). - Network policies/DNS prevent the worker from reaching the Prefect API (can’t receive cancel events; polling may also fail). - Namespace/SA mismatch: worker is looking in the wrong namespace or using a SA without rights. Quick checks 1) Inspect the flow run — does it have an infrastructure_pid, and what is the current state/message?

prefect flow-run inspect <flow-run-id>

2) Make sure a worker is connected and healthy (logs should show normal polling and not repeated restarts). 3) Check RBAC on the worker’s ServiceAccount for your target namespace:

kubectl auth can-i delete jobs   --as=system:serviceaccount:<ns>:<sa> -n <ns>
kubectl auth can-i delete pods   --as=system:serviceaccount:<ns>:<sa> -n <ns>
kubectl auth can-i list pods     --as=system:serviceaccount:<ns>:<sa> -n <ns>

If these return “no”, granting delete/list on jobs/pods will allow cancellation to complete. 4) Look at worker logs around the time you clicked Cancel. With DEBUG enabled (

PREFECT_LOGGING_LEVEL=DEBUG

), you should see it observe the Cancelling state and attempt cleanup; if it can’t find or delete the job, the error will be there. 5) Verify the namespace in your work pool/job template matches where the job would be created and where the SA has rights. 6) If you only ever saw “Worker KubernetesWorker submitting flow run” on submission, confirm whether

infrastructure_pid

was ever set on that run. If it’s missing but the run is in Cancelling, that’s a strong indicator the worker can’t complete cancellation. If you can share: - The output of

prefect flow-run inspect <id>

(especially

state

,

state_details

, and

infrastructure_pid

) - The worker’s namespace/ServiceAccount - Results of the

kubectl auth can-i delete jobs/pods

checks …I can help pinpoint whether this is a worker connectivity issue, RBAC, or an infra_pid/scoping problem. References - Prefect docs: Prefect 3.x docs (see Workers and Kubernetes integration sections)

Yes. In Prefect 3.x you can inspect and preview work pools/queues both via CLI and the UI. CLI options - List work pools

prefect work-pool ls

- Inspect a work pool (includes its queues and settings)

prefect work-pool inspect <pool-name>

- Preview all scheduled work across all queues in a pool

prefect work-pool preview <pool-name> --hours 24

- List work queues

prefect work-queue ls

- Inspect a specific queue

prefect work-queue inspect <queue-name> --pool <pool-name>

- Preview scheduled work for a queue

prefect work-queue preview <queue-name> --pool <pool-name> --hours 24

- Read runs currently in a queue (artificial poll)

prefect work-queue read-runs <queue-name> --pool <pool-name>

Note: this triggers an artificial poll of the queue on the server. It’s safe and does not submit the runs, but it will update poll metadata; avoid spamming it in production. UI options - Work Pools page → click your pool → Queues tab: see all queues, their status/concurrency, and preview upcoming scheduled work. - Runs page: filter by Work Pool and Work Queue to see the runs routed to a specific queue. If you share your pool/queue names, I can give exact command examples.

Great question. In Prefect 3.x with a KubernetesWorker, the model is pull-based: workers poll the Prefect API for work from their work pool; the work pool never “pushes” runs to workers. End-to-end workflow (start of a flow run) - Deployment and schedule - You create a Deployment assigned to a Work Pool (and usually a specific Work Queue). The server’s scheduler creates Flow Runs in Scheduled state. - Worker polling (pull model) - Your KubernetesWorker pod keeps a heartbeat and polls its Work Pool every few seconds (defaults: prefetch ≈10s ahead, poll ≈15s; configurable via env). - The server returns candidate Scheduled runs from that pool’s queues, honoring queue/pool concurrency limits. - Claiming the run - The worker proposes a Pending state for a candidate run. If orchestration accepts it, the worker has “claimed” the run. - You’ll see the log: “Worker KubernetesWorker submitting flow run …”. - Building the job - The worker merges the Work Pool’s base job template with deployment settings (env vars, image/entrypoint/source, resources, etc.). - For Kubernetes: it connects to the API, optionally creates a Secret for the Prefect API key, then creates a Job/Pod. - Submission and tracking - On success, the worker stores an infrastructure_pid on the flow run (e.g., namespace:job-name) so it can later cancel/clean it up. - The worker does not wait for completion; it returns to polling. - Inside the Job/Pod, the flow process connects directly to the Prefect API and updates state (Running → Completed/Failed, logs, etc.). - Cancellation (briefly) - If a run is Running and you click Cancel, the server sets it to Cancelling and the worker deletes the K8s Job/Pods using the infrastructure_pid. If no infra_pid (e.g., never submitted), the server skips Cancelling and marks it Cancelled directly. Key signals and limits - Concurrency limits can be set at the Work Pool and Work Queue levels; workers respect these when polling and claiming runs. - Important env vars for workers include polling/heartbeat tuning (e.g., PREFECT_WORKER_QUERY_SECONDS, PREFECT_WORKER_PREFETCH_SECONDS). Docs and references - Work Pools: Work Pools (concepts) - Workers (architecture, polling): Workers (concepts) - Kubernetes deployment guide: Kubernetes how-to - KubernetesWorker API (prefect-kubernetes): KubernetesWorker reference If you want, I can draw a quick sequence diagram for your specific setup (work pool name, queues, and how you’re packaging the flow code/image) and suggest best-practice settings for the worker deployment.