Andreas Nigg
08/17/2022, 11:27 AMprefect deployment build r_script_automation.py:r_script_automation --name retention_cohort_analysis_deployment -t k8s -sb gcs/gcs-prefect-stprage -ib kubernetes-job/generic-k8s-job -o cohort_deployment.yaml
However, as soon as I add infrastr. overrides, the flow stays in "pending" state forever
prefect deployment build r_script_automation.py:r_script_automation --name retention_cohort_analysis_deployment -t k8s -sb gcs/gcs-prefect-stprage -ib kubernetes-job/generic-k8s-job -o cohort_deployment.yaml --override image=europe-docker.pkg.dev/vol-at/rm-datateam-repository/r-script-automation:beta-23 --override env.GIT_PYTHON_REFRESH="quiet" --override env.GOOGLE_APPLICATION_CREDENTIALS="/google/.google-secret-key.json" --override env.CHROMIUM_FLAGS="--no-sandbox"
If I manually delete the infra_overrides-block from the deployment-yaml - the flow works again. So somehow this env.xyz - overrides are the problem. Can you point me to my mistake?
Edit: I tested overriding the "image" - this works. It really only fails when overriding the environment variables....Jeff Hale
08/17/2022, 12:23 PM--override env.PREFECT_LOGGING_LEVEL=DEBUG
Andreas Nigg
08/17/2022, 12:31 PMAndreas Nigg
08/17/2022, 12:31 PMAndreas Nigg
08/17/2022, 12:31 PMJeff Hale
08/17/2022, 2:17 PMinfra_overrides:
env.PREFECT_LOGGING_LEVEL: DEBUG
Andreas Nigg
08/17/2022, 3:26 PMAndreas Nigg
08/17/2022, 3:33 PM12:30:17.711 | INFO | prefect.agent - Submitting flow run 'bb691907-34d2-470e-ac38-85e8ae00c302'
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 193, in wrapper
return run_async_in_new_loop(async_fn, *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 140, in run_async_in_new_loop
return anyio.run(partial(__fn, *args, **kwargs))
File "/usr/local/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 70, in run
return asynclib.run(func, *args, **backend_options)
File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 292, in run
return native_run(wrapper(), debug=debug)
File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
return future.result()
File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
return await func(*args)
File "/usr/local/lib/python3.9/site-packages/prefect/cli/agent.py", line 104, in start
await critical_service_loop(
File "/usr/local/lib/python3.9/site-packages/prefect/agent.py", line 271, in __aexit__
await self.shutdown(*exc_info)
File "/usr/local/lib/python3.9/site-packages/prefect/agent.py", line 260, in shutdown
await self.task_group.__aexit__(*exc_info)
File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
raise exceptions[0]
File "/usr/local/lib/python3.9/site-packages/prefect/agent.py", line 182, in submit_run
infrastructure = await self.get_infrastructure(flow_run)
File "/usr/local/lib/python3.9/site-packages/prefect/agent.py", line 163, in get_infrastructure
data = data[field]
KeyError: 'env'
Andreas Nigg
08/17/2022, 5:30 PMAndreas Nigg
08/17/2022, 5:35 PM