https://prefect.io logo
Join Slack
Powered by
Hello Prefect community! Not sure if this is suppo...
# ask-community
c
Hello Prefect community! Not sure if this is supposed to be a feature or a bug...but all our k8s deployed flows started failing the last two days with this error on the K8s Agent (v2.3.2): FIXED: Specified`command` arg explicitly as 
[ "python", "-m", "prefect.engine" ]
for 
KubernetesJob
Copy code
18:49:52.809 | INFO    | prefect.agent - Submitting flow run 'faa2ebac-ee94-4384-9a4f-16ad28d6fa7e'
....  
  File "/usr/local/lib/python3.9/site-packages/prefect/blocks/core.py", line 171, in __init__
    super().__init__(*args, **kwargs)
  File "pydantic/main.py", line 342, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for KubernetesJob
command
  none is not an allowed value (type=type_error.none.not_allowed)
✅ 1
This is the KubernetesJob infra block we are associated with all deployments:
Copy code
k8s_container = {
    "name": "prefect-job",
    "env": [
        {
            "name": "SECRET_API_KEY",
            "valueFrom": {
                "secretKeyRef": {
                    "name": "secret-auth",
                    "key": "password",
                }
            },
        }
    ]
}

KubernetesJob(
    namespace="default",
    image=f"{os.environ['AWS_ECR_URL']}/prefect",
    image_pull_policy=KubernetesImagePullPolicy.ALWAYS,
    # Base job template:
    # <https://github.com/PrefectHQ/prefect/blob/main/src/prefect/infrastructure/kubernetes.py#L142>
    job={
        "apiVersion": "batch/v1",
        "kind": "Job",
        "metadata": {"labels": {}},
        "spec": {
            "ttlSecondsAfterFinished": 1800,
            "template": {
                "spec": {
                    "restartPolicy": "Never",
                    "parallelism": 1,
                    "completions": 1,
                    "containers": [k8s_container],
                    "affinity": {
                        "nodeAffinity": {
                            "preferredDuringSchedulingIgnoredDuringExecution": [
                                {
                                    "weight": 1,
                                    "preference": {
                                        "matchExpressions": [
                                            {
                                                "key": "nodegroup-name",
                                                "operator": "In",
                                                "values": ["compute"],
                                            }
                                        ]
                                    },
                                }
                            ]
                        }
                    },
                }
            },
        },
    },
    stream_output=True,
    pod_watch_timeout_seconds=60,
    job_watch_timeout_seconds=60,
)
Did something change over the last two days regarding how the K8s Agent creates new k8s jobs for a deployed flow according to 
KubernetesJob
block?
Ahh I noticed that Prefect v2.3.2 does not automatically fill in 
command
if missing, but v2.2.0 does. Fixed after specifying the command explicitly!
👍 2
a
Thanks @Chris L. let us know if there is anything we can do to help!
n
I just ran into this too, thanks for clarifying
8 Views
Open in Slack
PreviousNext
Powered by