hi all, I am running `prefect docker agent start....
# prefect-server
hi all, I am running
prefect docker agent start..
with some labels on my local machine, and am running into this error that I am unable to figure out. I have registered my flow, and it shows up in Prefect UI, but when I run that flow, it goes into Submitted State, and the logs of prefect docker agent show this error:
Copy code
docker.errors.NotFound: 404 Client Error for <http+docker://localhost/v1.41/containers/4675972bf8cb3506f9edd11788ac666dc61cef7d896c2e5367f38e9a9f3fdea2/json>: Not Found ("No such container: 4675972bf8cb3506f9edd11788ac666dc61cef7d896c2e5367f38e9a9f3fdea2"
discourse 1
Hi @Vishal, what is your Storage for the Flow? Can I see how you set up DockerRun?
this is how it shows up in UI for that flow:
What container are you using for DockerRun?
so I have this in my compose file, and i run
docker-compose run extract_metadata
which gets me into a container. I do
python flow.py --register
and whatever labels I get
Copy code
  image: my_image:latest
    - "./flow.py:/app/flow.py"
    - "../../local_configs/:/app/.prefect/"
    - ~/.env
    - PREFECT__SERVER__HOST=<http://host.docker.internal>
I run this after that:
Copy code
prefect agent docker start \
  -n "$(hostname) Docker Agent" \
  --label "6b7d73099ee0" \
  --label "env:dev" \
  --label "type:worker-flow" \
  --label "run-config:docker" \
  --volume "$(pwd)/flows/local_configs/flows/:/app/.prefect/flows/" \
A bit confused on the deployment here. Are you trying to run a flow with the
so that
container is used to register flow
And also to run the agent? Or the agent is outside?
the agent is outside
I see, can I see the Flow RunConfig or did you set any in
this is what my flow config looks like in
. we have a class behind the scenes that accepts that and needs
for DockerRun
Copy code
    "name": "my-flow",
    "run_config_type": "docker",
    "project_name": "data-eng-utilities",
    "is_orchestrator_flow": False,
    "image": "my-image:latest",
     "environment": "dev'

Do you have a longer traceback of the error?
Also does that container exist? Seems like the agent made a container to deploy the flow and then couldn’t find it to run.
Copy code
[2022-03-02 17:46:44,766] INFO - NYMBPEG4Q3MD6R Docker Agent | Registering agent...
[2022-03-02 17:46:44,792] INFO - NYMBPEG4Q3MD6R Docker Agent | Registration successful!

 ____            __           _        _                    _
|  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
| |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
|  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
|_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|

[2022-03-02 17:46:44,803] INFO - NYMBPEG4Q3MD6R Docker Agent | Starting DockerAgent with labels ['5c6b27197383', 'env:dev', 'run-config:docker', 'type:worker-flow']
[2022-03-02 17:46:44,803] INFO - NYMBPEG4Q3MD6R Docker Agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2022-03-02 17:46:44,803] INFO - NYMBPEG4Q3MD6R Docker Agent | Waiting for flow runs...
[2022-03-02 17:47:00,501] INFO - NYMBPEG4Q3MD6R Docker Agent | Deploying flow run a761733a-3a71-4b81-a461-e4a36d3490a6 to execution environment...
[2022-03-02 17:47:00,874] INFO - NYMBPEG4Q3MD6R Docker Agent | Completed deployment of flow run a761733a-3a71-4b81-a461-e4a36d3490a6
Process Process-1:
Traceback (most recent call last):
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/client.py", line 268, in _raise_for_status
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: <http+docker://localhost/v1.41/containers/0bee74f53f564b89978cd8daab062cf8f2a423a2b5c793c0134c64b122a7b9e6/json>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/prefect/agent/docker/agent.py", line 33, in _stream_container_logs
    for log in client.logs(container=container_id, stream=True, follow=True):
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/container.py", line 869, in logs
    output = self._get_result(container, stream, res)
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/client.py", line 460, in _get_result
    return self._get_result_tty(stream, res, self._check_is_tty(container))
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/client.py", line 456, in _check_is_tty
    cont = self.inspect_container(container)
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/container.py", line 771, in inspect_container
    return self._result(
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/client.py", line 274, in _result
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.NotFound: 404 Client Error for <http+docker://localhost/v1.41/containers/0bee74f53f564b89978cd8daab062cf8f2a423a2b5c793c0134c64b122a7b9e6/json>: Not Found ("No such container: 0bee74f53f564b89978cd8daab062cf8f2a423a2b5c793c0134c64b122a7b9e6")
so that container for some reason doesn’t seem to exist — i looked at all exited ones, and it doesn’t seem to be. And not sure why though
This is really weird because if the Agent did not point to the correct server, it wouldn’t pick up the flow. The only thing I can think of is the agent is incorrectly pointing to the Docker sock? It seems to be pointed somewhere else and trying to spin up a container somewhere else? I really don’t know
I think you are ending up running Docker in Docker. Check out this topic for more information I would recommend 1) registering the flow and 2) starting the Docker agent both in a local process to check if this is the issue
alternatively for Server with Docker compose, maybe you can try this approach here, I haven't tried that but many from the community found it helpful https://github.com/flavienbwk/prefect-docker-compose
also, you can inspect all docker containers and docker logs there to check if your Server can properly manage all the flow runs. Btw did you know that Prefect Cloud is free to use? you can have 20,000 free task runs entirely for free - no credit card required. Prefect Cloud is SIGNIFICANTLY easier to manage in such containerized setup because you have a publicly accessible API and you don't need to handle all the Docker networking overhead that causes you trouble right now - you can just connect your Docker agent via API key from Prefect cloud and you're done