Hi, has anyone successfully run connect prefect with an existing static dask cluster in kubernetes? I have setup a local dask cluster using the official dask helm chart with minikube and did port-forwarding so that prefect on my local machine can access the scheduler in the cluster. I can see that this connection was successful and the workers are receiving the task. But this is the error message:

Exception: PrefectHTTPStatusError("Client error '404 Not Found' for url '<http://ephemeral-prefect/api/task_runs/4d756c49-c449-4729-94a9-a7729d78e160/set_state>'\nResponse: {'exception_message': 'Task run with id 4d756c49-c449-4729-94a9-a7729d78e160 not found'}\nFor more information check: <https://httpstatuses.com/404>")

Hi @juana - is this error on the agent, or on the workers? It looks like you are running from the ephemeral client, and there is no callback from dask back to your local prefect? The client should be bi-directional communication with the scheduler, but the error itself seems like that task run doesn’t exist - do you have a more detailed log available?

Hi Christopher, sorry for the late response but I did figure it out shortly after that it was in ephemeral mode and i had to set the api url for all the workers. Thank you very much!