hi all I am running `prefect docker agent start ` with some Prefect Community #prefect-server

hi all, I am running `prefect docker agent start....

Vishal

03/02/2022, 4:51 PM

hi all, I am running

prefect docker agent start..

with some labels on my local machine, and am running into this error that I am unable to figure out. I have registered my flow, and it shows up in Prefect UI, but when I run that flow, it goes into Submitted State, and the logs of prefect docker agent show this error:

Copy code

docker.errors.NotFound: 404 Client Error for <http+docker://localhost/v1.41/containers/4675972bf8cb3506f9edd11788ac666dc61cef7d896c2e5367f38e9a9f3fdea2/json>: Not Found ("No such container: 4675972bf8cb3506f9edd11788ac666dc61cef7d896c2e5367f38e9a9f3fdea2"

discourse 1

Kevin Kho

03/02/2022, 4:52 PM

Hi @Vishal, what is your Storage for the Flow? Can I see how you set up DockerRun?

Vishal

03/02/2022, 5:03 PM

this is how it shows up in UI for that flow:

Kevin Kho

03/02/2022, 5:04 PM

What container are you using for DockerRun?

Vishal

03/02/2022, 5:08 PM

so I have this in my compose file, and i run

docker-compose run extract_metadata

which gets me into a container. I do

python flow.py --register

and whatever labels I get

Copy code

extract_metadata:
  image: my_image:latest
  volumes:
    - "./flow.py:/app/flow.py"
    - "../../local_configs/:/app/.prefect/"
  env_file:
    - ~/.env
  environment:
    - PREFECT__SERVER__HOST=<http://host.docker.internal>

I run this after that:

Copy code

prefect agent docker start \
  -n "$(hostname) Docker Agent" \
  --label "6b7d73099ee0" \
  --label "env:dev" \
  --label "type:worker-flow" \
  --label "run-config:docker" \
  --volume "$(pwd)/flows/local_configs/flows/:/app/.prefect/flows/" \
  --show-flow-logs

Kevin Kho

03/02/2022, 5:36 PM

A bit confused on the deployment here. Are you trying to run a flow with the

extract_metadata

container?

Vishal

03/02/2022, 5:37 PM

so that

extract_metadata

container is used to register flow

Kevin Kho

03/02/2022, 5:39 PM

And also to run the agent? Or the agent is outside?

Vishal

03/02/2022, 5:41 PM

the agent is outside

Kevin Kho

03/02/2022, 5:49 PM

I see, can I see the Flow RunConfig or did you set any in

flow.py

Vishal

03/02/2022, 5:55 PM

this is what my flow config looks like in

flow.py

. we have a class behind the scenes that accepts that and needs

image

for DockerRun

Copy code

FLOW_CONFIG = {
    "name": "my-flow",
    "run_config_type": "docker",
    "project_name": "data-eng-utilities",
    "is_orchestrator_flow": False,
    "image": "my-image:latest",
     "environment": "dev'

}

Kevin Kho

03/02/2022, 6:05 PM

Do you have a longer traceback of the error?

Kevin Kho

03/02/2022, 6:11 PM

Also does that container exist? Seems like the agent made a container to deploy the flow and then couldn’t find it to run.

Vishal

03/02/2022, 6:35 PM

Copy code

[2022-03-02 17:46:44,766] INFO - NYMBPEG4Q3MD6R Docker Agent | Registering agent...
[2022-03-02 17:46:44,792] INFO - NYMBPEG4Q3MD6R Docker Agent | Registration successful!

 ____            __           _        _                    _
|  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
| |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
|  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
|_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|
                                           |___/

[2022-03-02 17:46:44,803] INFO - NYMBPEG4Q3MD6R Docker Agent | Starting DockerAgent with labels ['5c6b27197383', 'env:dev', 'run-config:docker', 'type:worker-flow']
[2022-03-02 17:46:44,803] INFO - NYMBPEG4Q3MD6R Docker Agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2022-03-02 17:46:44,803] INFO - NYMBPEG4Q3MD6R Docker Agent | Waiting for flow runs...
[2022-03-02 17:47:00,501] INFO - NYMBPEG4Q3MD6R Docker Agent | Deploying flow run a761733a-3a71-4b81-a461-e4a36d3490a6 to execution environment...
[2022-03-02 17:47:00,874] INFO - NYMBPEG4Q3MD6R Docker Agent | Completed deployment of flow run a761733a-3a71-4b81-a461-e4a36d3490a6
Process Process-1:
Traceback (most recent call last):
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: <http+docker://localhost/v1.41/containers/0bee74f53f564b89978cd8daab062cf8f2a423a2b5c793c0134c64b122a7b9e6/json>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/prefect/agent/docker/agent.py", line 33, in _stream_container_logs
    for log in client.logs(container=container_id, stream=True, follow=True):
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/container.py", line 869, in logs
    output = self._get_result(container, stream, res)
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/client.py", line 460, in _get_result
    return self._get_result_tty(stream, res, self._check_is_tty(container))
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/client.py", line 456, in _check_is_tty
    cont = self.inspect_container(container)
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/container.py", line 771, in inspect_container
    return self._result(
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/client.py", line 274, in _result
    self._raise_for_status(response)
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.NotFound: 404 Client Error for <http+docker://localhost/v1.41/containers/0bee74f53f564b89978cd8daab062cf8f2a423a2b5c793c0134c64b122a7b9e6/json>: Not Found ("No such container: 0bee74f53f564b89978cd8daab062cf8f2a423a2b5c793c0134c64b122a7b9e6")

Vishal

03/02/2022, 6:35 PM

so that container for some reason doesn’t seem to exist — i looked at all exited ones, and it doesn’t seem to be. And not sure why though

Kevin Kho

03/02/2022, 6:40 PM

This is really weird because if the Agent did not point to the correct server, it wouldn’t pick up the flow. The only thing I can think of is the agent is incorrectly pointing to the Docker sock? It seems to be pointed somewhere else and trying to spin up a container somewhere else? I really don’t know

Anna Geller

03/02/2022, 7:30 PM

I think you are ending up running Docker in Docker. Check out this topic for more information I would recommend 1) registering the flow and 2) starting the Docker agent both in a local process to check if this is the issue

Anna Geller

03/02/2022, 7:32 PM

alternatively for Server with Docker compose, maybe you can try this approach here, I haven't tried that but many from the community found it helpful https://github.com/flavienbwk/prefect-docker-compose

Anna Geller

03/02/2022, 7:37 PM

also, you can inspect all docker containers and docker logs there to check if your Server can properly manage all the flow runs. Btw did you know that Prefect Cloud is free to use? you can have 20,000 free task runs entirely for free - no credit card required. Prefect Cloud is SIGNIFICANTLY easier to manage in such containerized setup because you have a publicly accessible API and you don't need to handle all the Docker networking overhead that causes you trouble right now - you can just connect your Docker agent via API key from Prefect cloud and you're done

Anna Geller

03/02/2022, 7:38 PM

this guide explains how you can get started https://docs.prefect.io/orchestration/getting-started/quick-start.html

2 Views

Open in Slack

Previous Next