hi guys, I have this error in Prefect Agent

ERROR   | prefect.agent - Failed to submit flow run 'xxxx' to infrastructure.

how can I investigate the details?

If your agent running Prefect 2.6.7? Normally you would see a traceback accompanying this error message, but 2.6.7 had a bug that excluded tracebacks. It is fixed in 2.6.8.

yeah i’m on 2.6.7

Bumping it to 2.6.8 if you can would be an ideal first step. Upgrading probably won't fix whatever caused the error, but it will ensure you get a traceback, which should help determine what went wrong.

great i see the traceback now 🙂

Glad to hear it! Feel free to post again in this thread if the traceback doesn't help solve the problem.

now I see it’s

DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

wonder what’s missing here…

I've seen a similar error when my Docker daemon crashed or failed to start

i guess so, it seems in the environment there’s no docker installed, since I deployed Prefect Agent in K8s using

prefect kubernetes manifest agent

and it uses standard

prefecthq/prefect:2.6.8-python3.9

image, seems there’s no docker in this image

I don't think it in included in any of our base images. As far as I know, running Docker inside a container isn't something we officially support/test for. That's not to say it won't work; I can't speak to it because I haven't tried it and I don't see much community discussion or docs mentioning it. The only way I've done it is by sharing the host's Docker socket with the container, and that worked well enough but that would probably not work on K8s.

it’s a bit strange that there’s not much discussions on this? How’d you guys then run the flow with the DockerContainer Block? wouldn’t the Agent execute the

docker

command? did I do something wrong?

I don't think you did anything wrong; it's just that most users running the

DockerContainer

block do it on a VM.

KubernetesJob

is more common on K8s, which is probably why we don't see a lot of Docker-in-Docker discussion. To answer your other question, when use use

DockerContainer

, the agent doesn't run any Docker commands; instead, it uses the Docker SDK for Python to talk directly to Docker's API

maybe i should also look into

KubernetesJob

😄

It sounds like it would be a good fit since you're already on K8s. It'll be more scalable because every flow run gets its own Kubernetes job.

so I was messing around with KubernetesJob, realize some points: • wanted to add some properties to the base job manifest, not sure how to edit the

customization

, are there any guides? • then i used

prefect kubernetes manifest flow-run-job

and copy pasted the manifest with my own inline changes • then i wanted to delete the wrongly configured

customization

(i know it’s wrong because the Agent won’t start it and gives me JsonPatch error), so i emptied the textbox in the UI, click save, the customization is still there, so i can’t delete it • then i want to delete the whole block and create a new one, i realize that after deleting the block, my deployments which depends on the block is also gone… is this a bug? does it make sense?