Hi I m trying to use the Docker storage method The image get Prefect Community #ask-community

Hi, I'm trying to use the Docker storage method. ...

flavienbwk

04/27/2021, 4:39 PM

Hi, I'm trying to use the Docker storage method. The image gets correctly saved in my registry but I get the following error in my agent :

Copy code

docker.errors.APIError: 400 Client Error for <http+docker://localhost/v1.40/containers/create>: Bad Request ("invalid IP address in add-host: """)

Could anyone help ? Thanks

flavienbwk

04/27/2021, 4:39 PM

Full error :

Copy code

agent_1  |  ____            __           _        _                    _
agent_1  | |  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
agent_1  | | |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
agent_1  | |  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
agent_1  | |_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|
agent_1  |                                            |___/
agent_1  | 
agent_1  | [2021-04-27 16:37:13,978] INFO - 21c3a5fb-f260-4b6e-a3c6-7257a1406b81 | Starting DockerAgent with labels ['development']
agent_1  | [2021-04-27 16:37:13,978] INFO - 21c3a5fb-f260-4b6e-a3c6-7257a1406b81 | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
agent_1  | [2021-04-27 16:37:13,978] INFO - 21c3a5fb-f260-4b6e-a3c6-7257a1406b81 | Agent connecting to the Prefect API at <http://172.17.0.1:4200>
agent_1  | [2021-04-27 16:37:13,986] INFO - 21c3a5fb-f260-4b6e-a3c6-7257a1406b81 | Waiting for flow runs...
agent_1  | [2021-04-27 16:37:15,625] INFO - 21c3a5fb-f260-4b6e-a3c6-7257a1406b81 | Found 1 flow run(s) to submit for execution.
agent_1  | [2021-04-27 16:37:15,693] INFO - 21c3a5fb-f260-4b6e-a3c6-7257a1406b81 | Deploying flow run c3e834e1-0e02-4e15-840b-dd954fd51365
agent_1  | [2021-04-27 16:37:15,694] INFO - 21c3a5fb-f260-4b6e-a3c6-7257a1406b81 | Pulling image 172.17.0.1:5000/30b93990-75a8-44b9-a900-e7f7083d883c:2021-04-27t16-31-36-020822-00-00...
agent_1  | [2021-04-27 16:37:15,704] INFO - 21c3a5fb-f260-4b6e-a3c6-7257a1406b81 | Successfully pulled image 172.17.0.1:5000/30b93990-75a8-44b9-a900-e7f7083d883c:2021-04-27t16-31-36-020822-00-00...
agent_1  | [2021-04-27 16:37:15,715] ERROR - 21c3a5fb-f260-4b6e-a3c6-7257a1406b81 | Logging platform error for flow run c3e834e1-0e02-4e15-840b-dd954fd51365
agent_1  | [2021-04-27 16:37:15,764] ERROR - 21c3a5fb-f260-4b6e-a3c6-7257a1406b81 | Error while deploying flow
agent_1  | Traceback (most recent call last):
agent_1  |   File "/usr/lib/python3.8/site-packages/docker/api/client.py", line 268, in _raise_for_status
agent_1  |     response.raise_for_status()
agent_1  |   File "/usr/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
agent_1  |     raise HTTPError(http_error_msg, response=self)
agent_1  | requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: <http+docker://localhost/v1.40/containers/create>
agent_1  | 
agent_1  | During handling of the above exception, another exception occurred:
agent_1  | 
agent_1  | Traceback (most recent call last):
agent_1  |   File "/usr/lib/python3.8/site-packages/prefect/agent/agent.py", line 414, in deploy_and_update_flow_run
agent_1  |     deployment_info = self.deploy_flow(flow_run)
agent_1  |   File "/usr/lib/python3.8/site-packages/prefect/agent/docker/agent.py", line 445, in deploy_flow
agent_1  |     container = self.docker_client.create_container(
agent_1  |   File "/usr/lib/python3.8/site-packages/docker/api/container.py", line 430, in create_container
agent_1  |     return self.create_container_from_config(config, name)
agent_1  |   File "/usr/lib/python3.8/site-packages/docker/api/container.py", line 441, in create_container_from_config
agent_1  |     return self._result(res, True)
agent_1  |   File "/usr/lib/python3.8/site-packages/docker/api/client.py", line 274, in _result
agent_1  |     self._raise_for_status(response)
agent_1  |   File "/usr/lib/python3.8/site-packages/docker/api/client.py", line 270, in _raise_for_status
agent_1  |     raise create_api_error_from_http_exception(e)
agent_1  |   File "/usr/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
agent_1  |     raise cls(e, response=response, explanation=explanation)
agent_1  | docker.errors.APIError: 400 Client Error for <http+docker://localhost/v1.40/containers/create>: Bad Request ("invalid IP address in add-host: """)

flavienbwk

04/27/2021, 4:40 PM

Registering flow this way :

Copy code

with Flow(
    "Get Paris' weather",
    storage=Docker(
        base_url="unix:///var/run/docker.sock",
        registry_url="172.17.0.1:5000",
        base_image="172.17.0.1:5000/weather/base_image:latest",
        ignore_healthchecks=True
    ),
) as flow:
    woeid = get_woeid("Paris")
    weather_data = get_weather(woeid)

Kevin Kho

04/27/2021, 4:42 PM

Hi @flavienbwk! ~~This looks like the agent in unable to pull from the registry right? Do you have some authentication for that registry or did you open it for internet traffic to download?~~

Zanie

04/27/2021, 4:47 PM

Hey @flavienbwk -- it looks like an issue with the auto-generated host-config ip, perhaps from this line? https://github.com/PrefectHQ/prefect/blob/master/src/prefect/agent/docker/agent.py#L422

Zanie

04/27/2021, 4:48 PM

Perhaps you could disable the

docker_interface

setting and see if that resolves it to narrow things down?

Zanie

04/27/2021, 4:51 PM

There's an open issue at https://github.com/PrefectHQ/prefect/issues/2781

👍 1

Zanie

04/27/2021, 4:52 PM

It'd be helpful if we could narrow down how this ip is empty so we can add a more helpful error message here

Zanie

04/27/2021, 4:52 PM

Could you try running the commands that https://github.com/PrefectHQ/prefect/blob/master/src/prefect/utilities/docker_util.py#L8 would execute?

flavienbwk

04/27/2021, 4:56 PM

Thank you @Kevin Kho, @Zanie. I am exactly in the case of a Docker in Docker configuration as mentionned by the comment of the

docker_interface

option. Here is the result of the following command (without the --no-docker-interface):

Copy code

administrateur@mycomputer:~$ docker exec -it 61b6dd sh
/ # python3
Python 3.8.8 (default, Mar 15 2021, 13:12:20) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import prefect
>>> prefect.utilities.docker_util.get_docker_ip()
''
>>>

Zanie

04/27/2021, 4:59 PM

That'll do it! We can add some better error handling in there but I'd love to determine where this is failing -- what's the output of

ip route

Zanie

04/27/2021, 5:00 PM

(and

ip route | grep docker0

)

flavienbwk

04/27/2021, 5:01 PM

BTW, it works with

--no-docker-interface

😄 Outputs of the commands :

Copy code

administrateur@mycomputer:~$ docker exec -it 357b sh
/ # ip route
default via 172.25.0.1 dev eth0 
172.25.0.0/16 dev eth0 proto kernel scope link src 172.25.0.2 
/ # ip route | grep docker0
/ #

Zanie

04/27/2021, 5:11 PM

https://github.com/PrefectHQ/prefect/pull/2328 appears to be just a workaround

Zanie

04/27/2021, 5:11 PM

I think we can just remove it entirely haha

flavienbwk

04/27/2021, 5:13 PM

I guess yes but I would not see the side effects 😛

flavienbwk

04/27/2021, 5:13 PM

Thanks for your help !

4 Views

Open in Slack

Previous Next