Hey folks, we are trying to move our task execution from ECS to EKS but running into this error as the agent tries to roll out pods for the job:

TypeError: '<' not supported between instances of 'NoneType' and 'datetime.datetime'
[2021-05-17 23:14:44,507] ERROR - agent | Error while managing existing k8s jobs
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 383, in heartbeat
    self.manage_jobs()
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 236, in manage_jobs
    pod_name
TypeError: '<' not supported between instances of 'NoneType' and 'datetime.datetime'

we’re using the config generated from the prefect cli for kubernetes agent. Any idea on how to work resolve it?

Hey @Walee -- this looks like a bug where events are missing timestamps. I've opened a fix at https://github.com/PrefectHQ/prefect/pull/4544 if you want to give it a try.

@Zanie thanks a bunch for putting it together so quickly! I’m giving it a try. Gonna report back soon

Glad it works! There'll be a published image on merge then it'll be in the next release which should be roughly in the next week.