Hey everyone, I keep getting very weird errors when trying to scale my workloads in a Kubernetes cluster. Large amounts of flow runs fail with the following error:

Encountered exception during execution:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1103, in begin_task_run
    return await orchestrate_task_run(
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1186, in orchestrate_task_run
    state = await propose_state(
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1484, in propose_state
    raise prefect.exceptions.Abort(response.details.reason)
prefect.exceptions.Abort: This run has already terminated.

This really tells me absolutely nothing, anyone have an idea maybe? A second error that I see again and again is this one:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/anyio/_core/_sockets.py", line 186, in connect_tcp
    addr_obj = ip_address(remote_host)
  File "/usr/local/lib/python3.10/ipaddress.py", line 54, in ip_address
    raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 address')
ValueError: 'api.prefect.cloud' does not appear to be an IPv4 or IPv6 address

Which again seems super cryptic to me 😅 Any hints to where I could start digging would be greatly appreciated, thanks 🙂

Hi Robin, does not appear to be an ipv4 address could indicate a dns error. it is valid:

nslookup api.prefect.cloud
Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
Name:	api.prefect.cloud
Address: 34.111.36.90

Regarding your first issue, it seems like you have duplicate flows attempting to run. Do you have just a single agent, or multiple agents? How are flows tagged and submitted for work?

Which task runner are you using?