This issue was introduced after update from Prefect server

0.13.19

to

0.14.4

. It's about the Kubernetes agent and how it creates the kubernetes job yaml specification. So this is the critical part of the yaml specification we have in `0.13.19`:

spec:
      containers:
      - args:
        - prefect execute flow-run
        command:
        - /bin/sh
        - -c

And this is the same part in

0.14.4

spec:
      containers:
      - args:
        - prefect
        - execute
        - flow-run
        command:
        - /bin/sh
        - -c

This is a problem for us, as we use vault, which injects some initContainers that load secrets and in the end end up running this command in the pod. Again first the working version

0.13.19

- args:
    - /bin/sh
    - -c
    - prefect execute flow-run
    command:
    - /vault/vault-env

and now the broken version

0.14.4

- args:
    - /bin/sh
    - -c
    - prefect
    - execute
    - flow-run
    command:
    - /vault/vault-env

The version

0.14.4

doesn't work, because it runs

/bin/sh -c prefect

which only prints help message and doesn't actually execute the flow.

Never mind, found the issue probably

We're using custom

job_template.yaml

which is based on the one in Prefect https://github.com/PrefectHQ/prefect/blob/0.14.4/src/prefect/agent/kubernetes/job_template.yaml. I didn't notice that the

command

was removed which was causing these issues.