https://prefect.io logo
Powered by
Title
v

Vishal

03/02/2022, 4:51 PM
hi all, I am running 
prefect docker agent start..
with some labels on my local machine, and am running into this error that I am unable to figure out. I have registered my flow, and it shows up in Prefect UI, but when I run that flow, it goes into Submitted State, and the logs of prefect docker agent show this error: 
docker.errors.NotFound: 404 Client Error for <http+docker://localhost/v1.41/containers/4675972bf8cb3506f9edd11788ac666dc61cef7d896c2e5367f38e9a9f3fdea2/json>: Not Found ("No such container: 4675972bf8cb3506f9edd11788ac666dc61cef7d896c2e5367f38e9a9f3fdea2"
:discourse: 1
k

Kevin Kho

03/02/2022, 4:52 PM
Hi @Vishal, what is your Storage for the Flow? Can I see how you set up DockerRun?
v

Vishal

03/02/2022, 5:03 PM
this is how it shows up in UI for that flow:
k

Kevin Kho

03/02/2022, 5:04 PM
What container are you using for DockerRun?
v

Vishal

03/02/2022, 5:08 PM
so I have this in my compose file, and i run 
docker-compose run extract_metadata
which gets me into a container. I do 
python flow.py --register
and whatever labels I get 
extract_metadata:
  image: my_image:latest
  volumes:
    - "./flow.py:/app/flow.py"
    - "../../local_configs/:/app/.prefect/"
  env_file:
    - ~/.env
  environment:
    - PREFECT__SERVER__HOST=<http://host.docker.internal>
I run this after that: 
prefect agent docker start \
  -n "$(hostname) Docker Agent" \
  --label "6b7d73099ee0" \
  --label "env:dev" \
  --label "type:worker-flow" \
  --label "run-config:docker" \
  --volume "$(pwd)/flows/local_configs/flows/:/app/.prefect/flows/" \
  --show-flow-logs
k

Kevin Kho

03/02/2022, 5:36 PM
A bit confused on the deployment here. Are you trying to run a flow with the 
extract_metadata
container?
v

Vishal

03/02/2022, 5:37 PM
so that 
extract_metadata
container is used to register flow
k

Kevin Kho

03/02/2022, 5:39 PM
And also to run the agent? Or the agent is outside?
v

Vishal

03/02/2022, 5:41 PM
the agent is outside
k

Kevin Kho

03/02/2022, 5:49 PM
I see, can I see the Flow RunConfig or did you set any in 
flow.py
?
v

Vishal

03/02/2022, 5:55 PM
this is what my flow config looks like in 
flow.py
. we have a class behind the scenes that accepts that and needs 
image
for DockerRun 
FLOW_CONFIG = {
    "name": "my-flow",
    "run_config_type": "docker",
    "project_name": "data-eng-utilities",
    "is_orchestrator_flow": False,
    "image": "my-image:latest",
     "environment": "dev'

}
k

Kevin Kho

03/02/2022, 6:05 PM
Do you have a longer traceback of the error?
Also does that container exist? Seems like the agent made a container to deploy the flow and then couldn’t find it to run.
v

Vishal

03/02/2022, 6:35 PM
[2022-03-02 17:46:44,766] INFO - NYMBPEG4Q3MD6R Docker Agent | Registering agent...
[2022-03-02 17:46:44,792] INFO - NYMBPEG4Q3MD6R Docker Agent | Registration successful!

 ____            __           _        _                    _
|  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
| |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
|  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
|_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|
                                           |___/

[2022-03-02 17:46:44,803] INFO - NYMBPEG4Q3MD6R Docker Agent | Starting DockerAgent with labels ['5c6b27197383', 'env:dev', 'run-config:docker', 'type:worker-flow']
[2022-03-02 17:46:44,803] INFO - NYMBPEG4Q3MD6R Docker Agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2022-03-02 17:46:44,803] INFO - NYMBPEG4Q3MD6R Docker Agent | Waiting for flow runs...
[2022-03-02 17:47:00,501] INFO - NYMBPEG4Q3MD6R Docker Agent | Deploying flow run a761733a-3a71-4b81-a461-e4a36d3490a6 to execution environment...
[2022-03-02 17:47:00,874] INFO - NYMBPEG4Q3MD6R Docker Agent | Completed deployment of flow run a761733a-3a71-4b81-a461-e4a36d3490a6
Process Process-1:
Traceback (most recent call last):
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: <http+docker://localhost/v1.41/containers/0bee74f53f564b89978cd8daab062cf8f2a423a2b5c793c0134c64b122a7b9e6/json>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/prefect/agent/docker/agent.py", line 33, in _stream_container_logs
    for log in client.logs(container=container_id, stream=True, follow=True):
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/container.py", line 869, in logs
    output = self._get_result(container, stream, res)
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/client.py", line 460, in _get_result
    return self._get_result_tty(stream, res, self._check_is_tty(container))
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/client.py", line 456, in _check_is_tty
    cont = self.inspect_container(container)
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/container.py", line 771, in inspect_container
    return self._result(
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/client.py", line 274, in _result
    self._raise_for_status(response)
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.NotFound: 404 Client Error for <http+docker://localhost/v1.41/containers/0bee74f53f564b89978cd8daab062cf8f2a423a2b5c793c0134c64b122a7b9e6/json>: Not Found ("No such container: 0bee74f53f564b89978cd8daab062cf8f2a423a2b5c793c0134c64b122a7b9e6")
so that container for some reason doesn’t seem to exist — i looked at all exited ones, and it doesn’t seem to be. And not sure why though
k

Kevin Kho

03/02/2022, 6:40 PM
This is really weird because if the Agent did not point to the correct server, it wouldn’t pick up the flow. The only thing I can think of is the agent is incorrectly pointing to the Docker sock? It seems to be pointed somewhere else and trying to spin up a container somewhere else? I really don’t know
a

Anna Geller

03/02/2022, 7:30 PM
I think you are ending up running Docker in Docker. Check out this topic for more information I would recommend 1) registering the flow and 2) starting the Docker agent both in a local process to check if this is the issue
alternatively for Server with Docker compose, maybe you can try this approach here, I haven't tried that but many from the community found it helpful https://github.com/flavienbwk/prefect-docker-compose
also, you can inspect all docker containers and docker logs there to check if your Server can properly manage all the flow runs. Btw did you know that Prefect Cloud is free to use? you can have 20,000 free task runs entirely for free - no credit card required. Prefect Cloud is SIGNIFICANTLY easier to manage in such containerized setup because you have a publicly accessible API and you don't need to handle all the Docker networking overhead that causes you trouble right now - you can just connect your Docker agent via API key from Prefect cloud and you're done
this guide explains how you can get started https://docs.prefect.io/orchestration/getting-started/quick-start.html
2 Views
#prefect-serverJoin Slack
Powered by