Thread
#prefect-server
    v

    Vishal

    6 months ago
    hi all, I am running
    prefect docker agent start..
    with some labels on my local machine, and am running into this error that I am unable to figure out. I have registered my flow, and it shows up in Prefect UI, but when I run that flow, it goes into Submitted State, and the logs of prefect docker agent show this error:
    docker.errors.NotFound: 404 Client Error for <http+docker://localhost/v1.41/containers/4675972bf8cb3506f9edd11788ac666dc61cef7d896c2e5367f38e9a9f3fdea2/json>: Not Found ("No such container: 4675972bf8cb3506f9edd11788ac666dc61cef7d896c2e5367f38e9a9f3fdea2"
    Kevin Kho

    Kevin Kho

    6 months ago
    Hi @Vishal, what is your Storage for the Flow? Can I see how you set up DockerRun?
    v

    Vishal

    6 months ago
    this is how it shows up in UI for that flow:
    Kevin Kho

    Kevin Kho

    6 months ago
    What container are you using for DockerRun?
    v

    Vishal

    6 months ago
    so I have this in my compose file, and i run
    docker-compose run extract_metadata
    which gets me into a container. I do
    python flow.py --register
    and whatever labels I get
    extract_metadata:
      image: my_image:latest
      volumes:
        - "./flow.py:/app/flow.py"
        - "../../local_configs/:/app/.prefect/"
      env_file:
        - ~/.env
      environment:
        - PREFECT__SERVER__HOST=<http://host.docker.internal>
    I run this after that:
    prefect agent docker start \
      -n "$(hostname) Docker Agent" \
      --label "6b7d73099ee0" \
      --label "env:dev" \
      --label "type:worker-flow" \
      --label "run-config:docker" \
      --volume "$(pwd)/flows/local_configs/flows/:/app/.prefect/flows/" \
      --show-flow-logs
    Kevin Kho

    Kevin Kho

    6 months ago
    A bit confused on the deployment here. Are you trying to run a flow with the
    extract_metadata
    container?
    v

    Vishal

    6 months ago
    so that
    extract_metadata
    container is used to register flow
    Kevin Kho

    Kevin Kho

    6 months ago
    And also to run the agent? Or the agent is outside?
    v

    Vishal

    6 months ago
    the agent is outside
    Kevin Kho

    Kevin Kho

    6 months ago
    I see, can I see the Flow RunConfig or did you set any in
    flow.py
    ?
    v

    Vishal

    6 months ago
    this is what my flow config looks like in
    flow.py
    . we have a class behind the scenes that accepts that and needs
    image
    for DockerRun
    FLOW_CONFIG = {
        "name": "my-flow",
        "run_config_type": "docker",
        "project_name": "data-eng-utilities",
        "is_orchestrator_flow": False,
        "image": "my-image:latest",
         "environment": "dev'
    
    }
    Kevin Kho

    Kevin Kho

    6 months ago
    Do you have a longer traceback of the error?
    Also does that container exist? Seems like the agent made a container to deploy the flow and then couldn’t find it to run.
    v

    Vishal

    6 months ago
    [2022-03-02 17:46:44,766] INFO - NYMBPEG4Q3MD6R Docker Agent | Registering agent...
    [2022-03-02 17:46:44,792] INFO - NYMBPEG4Q3MD6R Docker Agent | Registration successful!
    
     ____            __           _        _                    _
    |  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
    | |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
    |  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
    |_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|
                                               |___/
    
    [2022-03-02 17:46:44,803] INFO - NYMBPEG4Q3MD6R Docker Agent | Starting DockerAgent with labels ['5c6b27197383', 'env:dev', 'run-config:docker', 'type:worker-flow']
    [2022-03-02 17:46:44,803] INFO - NYMBPEG4Q3MD6R Docker Agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
    [2022-03-02 17:46:44,803] INFO - NYMBPEG4Q3MD6R Docker Agent | Waiting for flow runs...
    [2022-03-02 17:47:00,501] INFO - NYMBPEG4Q3MD6R Docker Agent | Deploying flow run a761733a-3a71-4b81-a461-e4a36d3490a6 to execution environment...
    [2022-03-02 17:47:00,874] INFO - NYMBPEG4Q3MD6R Docker Agent | Completed deployment of flow run a761733a-3a71-4b81-a461-e4a36d3490a6
    Process Process-1:
    Traceback (most recent call last):
      File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/client.py", line 268, in _raise_for_status
        response.raise_for_status()
      File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status
        raise HTTPError(http_error_msg, response=self)
    requests.exceptions.HTTPError: 404 Client Error: Not Found for url: <http+docker://localhost/v1.41/containers/0bee74f53f564b89978cd8daab062cf8f2a423a2b5c793c0134c64b122a7b9e6/json>
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
        self.run()
      File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 108, in run
        self._target(*self._args, **self._kwargs)
      File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/prefect/agent/docker/agent.py", line 33, in _stream_container_logs
        for log in client.logs(container=container_id, stream=True, follow=True):
      File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/utils/decorators.py", line 19, in wrapped
        return f(self, resource_id, *args, **kwargs)
      File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/container.py", line 869, in logs
        output = self._get_result(container, stream, res)
      File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/client.py", line 460, in _get_result
        return self._get_result_tty(stream, res, self._check_is_tty(container))
      File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/utils/decorators.py", line 19, in wrapped
        return f(self, resource_id, *args, **kwargs)
      File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/client.py", line 456, in _check_is_tty
        cont = self.inspect_container(container)
      File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/utils/decorators.py", line 19, in wrapped
        return f(self, resource_id, *args, **kwargs)
      File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/container.py", line 771, in inspect_container
        return self._result(
      File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/client.py", line 274, in _result
        self._raise_for_status(response)
      File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/api/client.py", line 270, in _raise_for_status
        raise create_api_error_from_http_exception(e)
      File "/Users/vkella/.pyenv/versions/3.8.0/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
        raise cls(e, response=response, explanation=explanation)
    docker.errors.NotFound: 404 Client Error for <http+docker://localhost/v1.41/containers/0bee74f53f564b89978cd8daab062cf8f2a423a2b5c793c0134c64b122a7b9e6/json>: Not Found ("No such container: 0bee74f53f564b89978cd8daab062cf8f2a423a2b5c793c0134c64b122a7b9e6")
    so that container for some reason doesn’t seem to exist — i looked at all exited ones, and it doesn’t seem to be. And not sure why though
    Kevin Kho

    Kevin Kho

    6 months ago
    This is really weird because if the Agent did not point to the correct server, it wouldn’t pick up the flow. The only thing I can think of is the agent is incorrectly pointing to the Docker sock? It seems to be pointed somewhere else and trying to spin up a container somewhere else? I really don’t know
    Anna Geller

    Anna Geller

    6 months ago
    I think you are ending up running Docker in Docker. Check out this topic for more information I would recommend 1) registering the flow and 2) starting the Docker agent both in a local process to check if this is the issue
    alternatively for Server with Docker compose, maybe you can try this approach here, I haven't tried that but many from the community found it helpful https://github.com/flavienbwk/prefect-docker-compose
    also, you can inspect all docker containers and docker logs there to check if your Server can properly manage all the flow runs. Btw did you know that Prefect Cloud is free to use? you can have 20,000 free task runs entirely for free - no credit card required. Prefect Cloud is SIGNIFICANTLY easier to manage in such containerized setup because you have a publicly accessible API and you don't need to handle all the Docker networking overhead that causes you trouble right now - you can just connect your Docker agent via API key from Prefect cloud and you're done