I see others might be experiencing similar issues when running Kubernetes workers, where jobs are started, complete, but stay in pending according to the Prefect Server. Here is what I have found: 1. When I trigger a Flow Run a job is created, in my case its called

skilled-rooster

. The logs in the UI get as far as saying running and the flow run seems to be stuck in Pending, see image below 2. At the same time, another Flow run is created,

hypnotic-caiman

, that doesn't seem to be attached to any Work pool, or anything. This Flow run executes and outputs what I would expect from the flow. 3. If I inspect the logs in k8s for the pod created by the

skilled-rooster

job I see that the first thing that happens when I run this, is the

hypnotic-caiman

flow run is created. My flow code is pretty much the example in the docs:

@flow(retries=3, retry_delay_seconds=5)
def get_repo_info():
    data = extract()
    stars, forks = tranform(data)
    load(stars, forks)


if __name__ == "__main__":
    get_repo_info()

And my

prefect.yaml

file looks like this:

name: flows
prefect-version: 2.14.4
build:
- prefect.deployments.steps.run_shell_script:
    id: get-commit-hash
    script: git rev-parse --short HEAD
    stream_output: false
push: null
pull: null

deployments:
- name: prefect_example
  schedule:
  work_pool:
    name: kubernetes
    job_variables:
      image: "<http://optimeeringexperiments.azurecr.io/prefect-example:{{|optimeeringexperiments.azurecr.io/prefect-example:{{> get-commit-hash.stdout }}"
      namespace: prefect
      image_pull_policy: Always
    work_queue_name:
  entrypoint: src/prefect/prefect_example/__main__.py:get_repo_info
  version:
  tags: []
  description:
  parameters: {}

Any ideas what might be going on here, I am not sure where to dig further.