https://prefect.io logo
Title
e

Emma Rizzi

04/06/2023, 8:10 AM
Hello, I'm trying to use a custom kubernetes-job block, every run gives this error :
data = data[field]
KeyError: 'env'
I reduced the job template to practically nothing custom here in case it was from my additionnal configuration, but still have this error. What am I doing wrong ?
1
last template I tried :
{
  "kind": "Job",
  "spec": {
    "template": {
      "spec": {
        "containers": [
          {
            "env": [],
            "name": "prefect-job"
          }
        ],
        "completions": 1,
        "parallelism": 1,
        "restartPolicy": "Never"
      },
      "metadata": {
        "labels": {}
      }
    }
  },
  "metadata": {
    "labels": {}
  },
  "apiVersion": "batch/v1"
}
full error stack :
07:58:25.456 | INFO    | prefect.agent - Submitting flow run 'ba1d889e-663f-4118-ab93-7591e5ce5c62'
07:58:26.281 | ERROR   | prefect.agent - Failed to get infrastructure for flow run 'ba1d889e-663f-4118-ab93-7591e5ce5c62'.
Traceback (most recent call last):
  File "/home/eouser/.local/lib/python3.8/site-packages/prefect/agent.py", line 440, in submit_run
    infrastructure = await self.get_infrastructure(flow_run)
  File "/home/eouser/.local/lib/python3.8/site-packages/prefect/agent.py", line 413, in get_infrastructure
    data = data[field]
KeyError: 'env'
r

redsquare

04/06/2023, 8:15 AM
assuming it deployed ok?
e

Emma Rizzi

04/06/2023, 8:15 AM
yes deployment is successful !
i've seen similar questions solved by adding the 'env' and 'labels' fields even if empty, I can't figure out whats missing now 😞
r

redsquare

04/06/2023, 8:18 AM
can you add a output = 'deployment_build_output.yaml' to the deployment and check the output
e

Emma Rizzi

04/06/2023, 8:22 AM
I got this:
###
### A complete description of a Prefect Deployment for flow 'check'
###
name: k8sjob
description: null
version: e433c7d8474dc025c33ba4ec3a690abe
# The work queue that will handle this deployment's runs
work_queue_name: default
work_pool_name: default-agent-pool
tags: []
parameters: {}
schedule: null
is_schedule_active: true
infra_overrides:
  env.EXTRA_PIP_PACKAGES: s3fs prefect-aws

###
### DO NOT EDIT BELOW THIS LINE
###
flow_name: check
manifest_path: null
infrastructure:
  type: kubernetes-job
  env: {}
  labels: {}
  name: null
  command: null
  image: prefecthq/prefect:2.8.7-python3.8
  namespace: prefect
  service_account_name: null
  image_pull_policy: null
  cluster_config: null
  job:
    kind: Job
    spec:
      template:
        spec:
          containers:
          - env: []
            name: prefect-job
          completions: 1
          parallelism: 1
          restartPolicy: Never
        metadata:
          labels: {}
    metadata:
      labels: {}
    apiVersion: batch/v1
  customizations: []
  job_watch_timeout_seconds: null
  pod_watch_timeout_seconds: 60
  stream_output: true
  finished_job_ttl: 21600
  _block_document_id: 80301a41-ae80-4a1d-a5d3-e8940e59a02a
  _block_document_name: eouser-job
  _is_anonymous: false
  block_type_slug: kubernetes-job
  _block_type_slug: kubernetes-job
storage:
  bucket_name: prefect
  minio_credentials: null
  aws_credentials: null
  basepath: ''
  endpoint_url: null
  credentials:
    aws_access_key_id: ###
    aws_secret_access_key: '**********'
    aws_session_token: null
    profile_name: null
    region_name: eu-west-1
    aws_client_parameters:
      api_version: null
      use_ssl: true
      verify: true
      verify_cert_path: null
      endpoint_url: ###
      config: null
    _block_document_id: 3b2f6b95-f722-4385-b0e7-ff84e9bfcc21
    _block_document_name: s3-credentials
    _is_anonymous: false
    block_type_slug: aws-credentials
  bucket_folder: ''
  _block_document_id: dbed56f4-48cd-49b1-9d23-93b45ca427ad
  _block_document_name: wekeo-s3
  _is_anonymous: false
  block_type_slug: s3-bucket
  _block_type_slug: s3-bucket
path: test/
entrypoint: flow.py:check
parameter_openapi_schema:
  title: Parameters
  type: object
  properties: {}
  required: null
  definitions: null
timestamp: '2023-04-06T08:20:26.898949+00:00'
r

redsquare

04/06/2023, 8:26 AM
looks fine to me, exactly the same as mine except the spec.template.metadata
hang on - how are you defining EXTRA_PIP_PACKAGES
mine is
infra_overrides:
env:
EXTRA_PIP_PACKAGES: s3fs datadog python-dotenv
e

Emma Rizzi

04/06/2023, 8:29 AM
right now i pass them in the build command :
prefect deployment build flow.py:check -n k8sjob -sb s3-bucket/wekeo-s3 -ib kubernetes-job/eouser-job --override env.EXTRA_PIP_PACKAGES="s3fs prefect-aws" -a
, i only have dependencies for S3 storage block. Everything worked until I added the custom infra block
whats different with your spec.template.metadata ?
r

redsquare

04/06/2023, 8:31 AM
I dont have it
in my base manifest
e

Emma Rizzi

04/06/2023, 8:34 AM
i had hopes, i removed it but still the same
KeyError: 'env'
😭
{
  "kind": "Job",
  "spec": {
    "template": {
      "spec": {
        "containers": [
          {
            "env": [],
            "name": "prefect-job"
          }
        ],
        "completions": 1,
        "parallelism": 1,
        "restartPolicy": "Never"
      }
    }
  },
  "metadata": {
    "labels": {}
  },
  "apiVersion": "batch/v1"
}
r

redsquare

04/06/2023, 8:34 AM
Its defo your overrides
e

Emma Rizzi

04/06/2023, 8:39 AM
gosh yes! I didn't understand your first message about dependencies, that was it!
--override env.EXTRA_PIP_PACKAGES
seems not compatible with infra block, i put the dependencies inside the job template and it worked! thanks a lot 🙂 now I'll add the actual custom kube config hopefully not breaking everything again 🤞
r

redsquare

04/06/2023, 8:40 AM
looking at the source it splits by a dot, so env.EXTRA_PIP_PACKAGES="s3fs.prefect-aws" should work
e

Emma Rizzi

04/06/2023, 8:41 AM
thanks, I'll next be building custom docker image including dependencies so that should fix it also 🙂
👍 1