Hi, we are observing following exception in prefect worker (in kubernetes). We don't use to see this exception before. Are we missing something?

UnboundLocalError: local variable 'configuration' referenced before assignment
18:00:03.690 | WARNING | prefect.worker.kubernetes.kubernetesworker 550d0424-d2d4-45dd-9e5a-a32d69a9786c - Flow run UUID('da62825e-bb01-4a80-b74e-fc203f8fec4f') cannot be cancelled by this worker: associated deployment UUID('38ae5af7-e26b-478d-a5a4-017899ef2919') does not exist.
18:00:03.690 | ERROR | prefect.flow_runs.worker - Encountered exception while killing infrastructure for flow run 'da62825e-bb01-4a80-b74e-fc203f8fec4f'. Flow run may not be cancelled.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/workers/base.py", line 668, in cancel_run
configuration=configuration,

what version of prefect and prefect-kubernetes is your worker on?

saw this on 2.14.9 and it was also OOMing, not updated to 2.14.18, and this exceptions we are seeing in new version as well

hey roshan, it seems like what happened is that a deployment was deleted while the flow run was executing, can you confirm if

deployment UUID('38ae5af7-e26b-478d-a5a4-017899ef2919')

is an existing deployment ID?

It doesn't exits, seems to be older one that's already deleted

i don't think you should be seeing this exception anymore because there should be no way to create new flow-runs from the deployment that was deleted. are you seeing new flow-runs have the same issue?

new flow runs are not having this issue but its flooding the log of worker with above exception

oh i see, the worker is continually trying to cancel the flow run and hitting the exception -- can you try deleting the flow-run using the CLI:

prefect flow-run delete da62825e-bb01-4a80-b74e-fc203f8fec4f

that should stop your worker from doing that

will try it and update you. Thanks

FYI I have a fix in progress to prevent this from ever happening over in this PR