https://prefect.io logo
Title
j

James Zhang

11/21/2022, 3:24 PM
hi guys, I have this error in Prefect Agent
ERROR   | prefect.agent - Failed to submit flow run 'xxxx' to infrastructure.
how can I investigate the details?
1
r

Ryan Peden

11/21/2022, 3:28 PM
If your agent running Prefect 2.6.7? Normally you would see a traceback accompanying this error message, but 2.6.7 had a bug that excluded tracebacks. It is fixed in 2.6.8.
j

James Zhang

11/21/2022, 3:29 PM
yeah i’m on 2.6.7
r

Ryan Peden

11/21/2022, 3:31 PM
Bumping it to 2.6.8 if you can would be an ideal first step. Upgrading probably won't fix whatever caused the error, but it will ensure you get a traceback, which should help determine what went wrong.
👍 1
j

James Zhang

11/21/2022, 3:32 PM
great i see the traceback now 🙂
r

Ryan Peden

11/21/2022, 3:34 PM
Glad to hear it! Feel free to post again in this thread if the traceback doesn't help solve the problem.
j

James Zhang

11/21/2022, 3:44 PM
now I see it’s
DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
wonder what’s missing here…
r

Ryan Peden

11/21/2022, 3:50 PM
I've seen a similar error when my Docker daemon crashed or failed to start
j

James Zhang

11/21/2022, 3:51 PM
i guess so, it seems in the environment there’s no docker installed, since I deployed Prefect Agent in K8s using
prefect kubernetes manifest agent
and it uses standard
prefecthq/prefect:2.6.8-python3.9
image, seems there’s no docker in this image
r

Ryan Peden

11/21/2022, 4:09 PM
I don't think it in included in any of our base images. As far as I know, running Docker inside a container isn't something we officially support/test for. That's not to say it won't work; I can't speak to it because I haven't tried it and I don't see much community discussion or docs mentioning it. The only way I've done it is by sharing the host's Docker socket with the container, and that worked well enough but that would probably not work on K8s.
j

James Zhang

11/21/2022, 4:28 PM
it’s a bit strange that there’s not much discussions on this? How’d you guys then run the flow with the DockerContainer Block? wouldn’t the Agent execute the
docker
command? did I do something wrong?
r

Ryan Peden

11/21/2022, 4:34 PM
I don't think you did anything wrong; it's just that most users running the
DockerContainer
block do it on a VM.
KubernetesJob
is more common on K8s, which is probably why we don't see a lot of Docker-in-Docker discussion. To answer your other question, when use use
DockerContainer
, the agent doesn't run any Docker commands; instead, it uses the Docker SDK for Python to talk directly to Docker's API
j

James Zhang

11/21/2022, 4:35 PM
maybe i should also look into
KubernetesJob
😄
r

Ryan Peden

11/21/2022, 4:40 PM
It sounds like it would be a good fit since you're already on K8s. It'll be more scalable because every flow run gets its own Kubernetes job.
j

James Zhang

11/21/2022, 10:16 PM
so I was messing around with KubernetesJob, realize some points: • wanted to add some properties to the base job manifest, not sure how to edit the
customization
, are there any guides? • then i used
prefect kubernetes manifest flow-run-job
and copy pasted the manifest with my own inline changes • then i wanted to delete the wrongly configured
customization
(i know it’s wrong because the Agent won’t start it and gives me JsonPatch error), so i emptied the textbox in the UI, click save, the customization is still there, so i can’t delete it • then i want to delete the whole block and create a new one, i realize that after deleting the block, my deployments which depends on the block is also gone… is this a bug? does it make sense?