flavienbwk

    flavienbwk

    1 year ago
    Hi, I'm trying to use the Docker storage method. The image gets correctly saved in my registry but I get the following error in my agent :
    docker.errors.APIError: 400 Client Error for <http+docker://localhost/v1.40/containers/create>: Bad Request ("invalid IP address in add-host: """)
    Could anyone help ? Thanks
    Full error :
    agent_1  |  ____            __           _        _                    _
    agent_1  | |  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
    agent_1  | | |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
    agent_1  | |  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
    agent_1  | |_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|
    agent_1  |                                            |___/
    agent_1  | 
    agent_1  | [2021-04-27 16:37:13,978] INFO - 21c3a5fb-f260-4b6e-a3c6-7257a1406b81 | Starting DockerAgent with labels ['development']
    agent_1  | [2021-04-27 16:37:13,978] INFO - 21c3a5fb-f260-4b6e-a3c6-7257a1406b81 | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
    agent_1  | [2021-04-27 16:37:13,978] INFO - 21c3a5fb-f260-4b6e-a3c6-7257a1406b81 | Agent connecting to the Prefect API at <http://172.17.0.1:4200>
    agent_1  | [2021-04-27 16:37:13,986] INFO - 21c3a5fb-f260-4b6e-a3c6-7257a1406b81 | Waiting for flow runs...
    agent_1  | [2021-04-27 16:37:15,625] INFO - 21c3a5fb-f260-4b6e-a3c6-7257a1406b81 | Found 1 flow run(s) to submit for execution.
    agent_1  | [2021-04-27 16:37:15,693] INFO - 21c3a5fb-f260-4b6e-a3c6-7257a1406b81 | Deploying flow run c3e834e1-0e02-4e15-840b-dd954fd51365
    agent_1  | [2021-04-27 16:37:15,694] INFO - 21c3a5fb-f260-4b6e-a3c6-7257a1406b81 | Pulling image 172.17.0.1:5000/30b93990-75a8-44b9-a900-e7f7083d883c:2021-04-27t16-31-36-020822-00-00...
    agent_1  | [2021-04-27 16:37:15,704] INFO - 21c3a5fb-f260-4b6e-a3c6-7257a1406b81 | Successfully pulled image 172.17.0.1:5000/30b93990-75a8-44b9-a900-e7f7083d883c:2021-04-27t16-31-36-020822-00-00...
    agent_1  | [2021-04-27 16:37:15,715] ERROR - 21c3a5fb-f260-4b6e-a3c6-7257a1406b81 | Logging platform error for flow run c3e834e1-0e02-4e15-840b-dd954fd51365
    agent_1  | [2021-04-27 16:37:15,764] ERROR - 21c3a5fb-f260-4b6e-a3c6-7257a1406b81 | Error while deploying flow
    agent_1  | Traceback (most recent call last):
    agent_1  |   File "/usr/lib/python3.8/site-packages/docker/api/client.py", line 268, in _raise_for_status
    agent_1  |     response.raise_for_status()
    agent_1  |   File "/usr/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
    agent_1  |     raise HTTPError(http_error_msg, response=self)
    agent_1  | requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: <http+docker://localhost/v1.40/containers/create>
    agent_1  | 
    agent_1  | During handling of the above exception, another exception occurred:
    agent_1  | 
    agent_1  | Traceback (most recent call last):
    agent_1  |   File "/usr/lib/python3.8/site-packages/prefect/agent/agent.py", line 414, in deploy_and_update_flow_run
    agent_1  |     deployment_info = self.deploy_flow(flow_run)
    agent_1  |   File "/usr/lib/python3.8/site-packages/prefect/agent/docker/agent.py", line 445, in deploy_flow
    agent_1  |     container = self.docker_client.create_container(
    agent_1  |   File "/usr/lib/python3.8/site-packages/docker/api/container.py", line 430, in create_container
    agent_1  |     return self.create_container_from_config(config, name)
    agent_1  |   File "/usr/lib/python3.8/site-packages/docker/api/container.py", line 441, in create_container_from_config
    agent_1  |     return self._result(res, True)
    agent_1  |   File "/usr/lib/python3.8/site-packages/docker/api/client.py", line 274, in _result
    agent_1  |     self._raise_for_status(response)
    agent_1  |   File "/usr/lib/python3.8/site-packages/docker/api/client.py", line 270, in _raise_for_status
    agent_1  |     raise create_api_error_from_http_exception(e)
    agent_1  |   File "/usr/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    agent_1  |     raise cls(e, response=response, explanation=explanation)
    agent_1  | docker.errors.APIError: 400 Client Error for <http+docker://localhost/v1.40/containers/create>: Bad Request ("invalid IP address in add-host: """)
    Registering flow this way :
    with Flow(
        "Get Paris' weather",
        storage=Docker(
            base_url="unix:///var/run/docker.sock",
            registry_url="172.17.0.1:5000",
            base_image="172.17.0.1:5000/weather/base_image:latest",
            ignore_healthchecks=True
        ),
    ) as flow:
        woeid = get_woeid("Paris")
        weather_data = get_weather(woeid)
    Kevin Kho

    Kevin Kho

    1 year ago
    Hi @flavienbwk! This looks like the agent in unable to pull from the registry right? Do you have some authentication for that registry or did you open it for internet traffic to download?
    Michael Adkins

    Michael Adkins

    1 year ago
    Hey @flavienbwk -- it looks like an issue with the auto-generated host-config ip, perhaps from this line? https://github.com/PrefectHQ/prefect/blob/master/src/prefect/agent/docker/agent.py#L422
    Perhaps you could disable the
    docker_interface
    setting and see if that resolves it to narrow things down?
    It'd be helpful if we could narrow down how this ip is empty so we can add a more helpful error message here
    flavienbwk

    flavienbwk

    1 year ago
    Thank you @Kevin Kho, @Michael Adkins. I am exactly in the case of a Docker in Docker configuration as mentionned by the comment of the
    docker_interface
    option. Here is the result of the following command (without the --no-docker-interface):
    administrateur@mycomputer:~$ docker exec -it 61b6dd sh
    / # python3
    Python 3.8.8 (default, Mar 15 2021, 13:12:20) 
    [GCC 9.3.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import prefect
    >>> prefect.utilities.docker_util.get_docker_ip()
    ''
    >>>
    Michael Adkins

    Michael Adkins

    1 year ago
    That'll do it! We can add some better error handling in there but I'd love to determine where this is failing -- what's the output of
    ip route
    (and
    ip route | grep docker0
    )
    flavienbwk

    flavienbwk

    1 year ago
    BTW, it works with
    --no-docker-interface
    😄 Outputs of the commands :
    administrateur@mycomputer:~$ docker exec -it 357b sh
    / # ip route
    default via 172.25.0.1 dev eth0 
    172.25.0.0/16 dev eth0 proto kernel scope link src 172.25.0.2 
    / # ip route | grep docker0
    / #
    Michael Adkins

    Michael Adkins

    1 year ago
    https://github.com/PrefectHQ/prefect/pull/2328 appears to be just a workaround
    I think we can just remove it entirely haha
    flavienbwk

    flavienbwk

    1 year ago
    I guess yes but I would not see the side effects 😛
    Thanks for your help !