Emma Rizzi
04/06/2023, 8:10 AMdata = data[field]
KeyError: 'env'
I reduced the job template to practically nothing custom here in case it was from my additionnal configuration, but still have this error. What am I doing wrong ?{
"kind": "Job",
"spec": {
"template": {
"spec": {
"containers": [
{
"env": [],
"name": "prefect-job"
}
],
"completions": 1,
"parallelism": 1,
"restartPolicy": "Never"
},
"metadata": {
"labels": {}
}
}
},
"metadata": {
"labels": {}
},
"apiVersion": "batch/v1"
}
07:58:25.456 | INFO | prefect.agent - Submitting flow run 'ba1d889e-663f-4118-ab93-7591e5ce5c62'
07:58:26.281 | ERROR | prefect.agent - Failed to get infrastructure for flow run 'ba1d889e-663f-4118-ab93-7591e5ce5c62'.
Traceback (most recent call last):
File "/home/eouser/.local/lib/python3.8/site-packages/prefect/agent.py", line 440, in submit_run
infrastructure = await self.get_infrastructure(flow_run)
File "/home/eouser/.local/lib/python3.8/site-packages/prefect/agent.py", line 413, in get_infrastructure
data = data[field]
KeyError: 'env'
redsquare
04/06/2023, 8:15 AMEmma Rizzi
04/06/2023, 8:15 AMredsquare
04/06/2023, 8:18 AMEmma Rizzi
04/06/2023, 8:22 AM###
### A complete description of a Prefect Deployment for flow 'check'
###
name: k8sjob
description: null
version: e433c7d8474dc025c33ba4ec3a690abe
# The work queue that will handle this deployment's runs
work_queue_name: default
work_pool_name: default-agent-pool
tags: []
parameters: {}
schedule: null
is_schedule_active: true
infra_overrides:
env.EXTRA_PIP_PACKAGES: s3fs prefect-aws
###
### DO NOT EDIT BELOW THIS LINE
###
flow_name: check
manifest_path: null
infrastructure:
type: kubernetes-job
env: {}
labels: {}
name: null
command: null
image: prefecthq/prefect:2.8.7-python3.8
namespace: prefect
service_account_name: null
image_pull_policy: null
cluster_config: null
job:
kind: Job
spec:
template:
spec:
containers:
- env: []
name: prefect-job
completions: 1
parallelism: 1
restartPolicy: Never
metadata:
labels: {}
metadata:
labels: {}
apiVersion: batch/v1
customizations: []
job_watch_timeout_seconds: null
pod_watch_timeout_seconds: 60
stream_output: true
finished_job_ttl: 21600
_block_document_id: 80301a41-ae80-4a1d-a5d3-e8940e59a02a
_block_document_name: eouser-job
_is_anonymous: false
block_type_slug: kubernetes-job
_block_type_slug: kubernetes-job
storage:
bucket_name: prefect
minio_credentials: null
aws_credentials: null
basepath: ''
endpoint_url: null
credentials:
aws_access_key_id: ###
aws_secret_access_key: '**********'
aws_session_token: null
profile_name: null
region_name: eu-west-1
aws_client_parameters:
api_version: null
use_ssl: true
verify: true
verify_cert_path: null
endpoint_url: ###
config: null
_block_document_id: 3b2f6b95-f722-4385-b0e7-ff84e9bfcc21
_block_document_name: s3-credentials
_is_anonymous: false
block_type_slug: aws-credentials
bucket_folder: ''
_block_document_id: dbed56f4-48cd-49b1-9d23-93b45ca427ad
_block_document_name: wekeo-s3
_is_anonymous: false
block_type_slug: s3-bucket
_block_type_slug: s3-bucket
path: test/
entrypoint: flow.py:check
parameter_openapi_schema:
title: Parameters
type: object
properties: {}
required: null
definitions: null
timestamp: '2023-04-06T08:20:26.898949+00:00'
redsquare
04/06/2023, 8:26 AMinfra_overrides:
env:
EXTRA_PIP_PACKAGES: s3fs datadog python-dotenv
Emma Rizzi
04/06/2023, 8:29 AMprefect deployment build flow.py:check -n k8sjob -sb s3-bucket/wekeo-s3 -ib kubernetes-job/eouser-job --override env.EXTRA_PIP_PACKAGES="s3fs prefect-aws" -a
, i only have dependencies for S3 storage block. Everything worked until I added the custom infra blockredsquare
04/06/2023, 8:31 AMEmma Rizzi
04/06/2023, 8:34 AMKeyError: 'env'
😭{
"kind": "Job",
"spec": {
"template": {
"spec": {
"containers": [
{
"env": [],
"name": "prefect-job"
}
],
"completions": 1,
"parallelism": 1,
"restartPolicy": "Never"
}
}
},
"metadata": {
"labels": {}
},
"apiVersion": "batch/v1"
}
redsquare
04/06/2023, 8:34 AMEmma Rizzi
04/06/2023, 8:39 AM--override env.EXTRA_PIP_PACKAGES
seems not compatible with infra block, i put the dependencies inside the job template and it worked!
thanks a lot 🙂 now I'll add the actual custom kube config hopefully not breaking everything again 🤞redsquare
04/06/2023, 8:40 AMEmma Rizzi
04/06/2023, 8:41 AM