Michael Hadorn
05/19/2021, 3:35 PMMay 19 17:30:35 <http://XXX.ch|XXX.ch> run_dev_docker_agent.sh[3306015]: requests.exceptions.ConnectionError: HTTPConnectionPool(host='host.docker.internal', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8572e5ecd0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
(-> full log in the thread.)
Before (<=0.14.17) everything was working.
On Windows it's working also with .19.
But not on:
Ubuntu 20.04
Docker version 20.10.3, build 48d30b5
docker-compose version 1.28.2, build unknown
Do you get some similar behaviours?Kevin Kho
05/19/2021, 3:59 PMMichael Hadorn
05/19/2021, 4:01 PMKevin Kho
05/19/2021, 4:01 PMZanie
05/19/2021, 4:25 PMMichael Hadorn
05/19/2021, 4:27 PMprefect agent docker start -l prefect-development --show-flow-logs
Kevin Kho
05/20/2021, 11:06 PMMichael Hadorn
05/21/2021, 10:32 AMprefecthq/prefect:0.14.19-python3.8
So it's Python 3.8.9Kevin Kho
05/21/2021, 12:20 PMMichael Hadorn
05/21/2021, 12:32 PMKevin Kho
05/21/2021, 2:25 PMMichael Hadorn
05/21/2021, 2:25 PMKevin Kho
05/21/2021, 2:25 PMZanie
05/21/2021, 2:26 PMMichael Hadorn
05/21/2021, 2:26 PMZanie
05/21/2021, 2:26 PMMichael Hadorn
05/21/2021, 2:26 PMZanie
05/21/2021, 2:27 PMprefect diagnostics
?Michael Hadorn
05/21/2021, 2:28 PM{
"config_overrides": {
"env": {
"db": {
"host": true,
"password": true,
"type": true,
"user": true
},
"name": true
},
"general": {
"dependency_valid_window_h": true,
"email": {
"email_from": true,
"email_to": true,
"enable": true,
"smtp_port": true,
"smtp_server": true,
"smtp_type": true
},
"flow_name": true,
"ignore_last_run_state": true,
"ignore_tables": true,
"n_workers": true,
"skip_already_done_tasks": true
},
"object": {
"fct_labor_result_xlab_": {
"delta_expr_ge": true
}
}
},
"env_vars": [],
"system_information": {
"platform": "Linux-5.4.0-73-generic-x86_64-with-glibc2.10",
"prefect_backend": "server",
"prefect_version": "0.14.19",
"python_version": "3.8.8"
}
}
Zanie
05/21/2021, 2:30 PMpip show click
Michael Hadorn
05/21/2021, 2:30 PMName: click
Version: 8.0.0
Summary: Composable command line interface toolkit
Home-page: <https://palletsprojects.com/p/click/>
Author: Armin Ronacher
Author-email: <mailto:armin.ronacher@active-4.com|armin.ronacher@active-4.com>
License: BSD-3-Clause
Location: /home/poc/miniconda3/envs/prefect/lib/python3.8/site-packages
Requires:
Required-by: prefect, distributed
Zanie
05/21/2021, 2:31 PMpip install click\<8.0
Michael Hadorn
05/21/2021, 2:34 PMZanie
05/21/2021, 2:34 PMMichael Hadorn
05/21/2021, 2:35 PMZanie
05/21/2021, 2:35 PMsrc/prefect/agent/docker/agent.py#L416
we set the containers to auto-remove host_config = {"auto_remove": True} # type: dict
-- if you set this to False
instead we should get the flow container to stick around after it fails (you can find the prefect installation location with pip show prefect
and modify this file)result = client.graphql(query)
shows that the error is happening when the container tries to reach the Server API. I'm presuming the API URL in the container is incorrect or something.Michael Hadorn
05/21/2021, 2:58 PM> docker ps --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
126f58926104 cdwh/cdwh-flow:prefect-development "tini -g -- entrypoi…" About a minute ago Exited (1) About a minute ago mini-toucanet
> docker exec -it 126f58926104 bash
Error response from daemon: Container 126f58926104416a4a253a640df7dab1a0750fef43e7195eeabb091112ab1cdb is not running
Zanie
05/21/2021, 3:12 PMdocker logs 669bc43012c3
and see if there's anything good in there?Michael Hadorn
05/21/2021, 3:14 PM...
requests.exceptions.ConnectionError: HTTPConnectionPool(host='host.docker.internal', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f53e35ff940>: Failed to establish a new connection: [Errno -2] Name or service not known'))
Zanie
05/21/2021, 3:14 PMdocker inspect 669bc43012c3
and share the Env
portion?Michael Hadorn
05/21/2021, 3:23 PM"Env": [
"PREFECT__LOGGING__LEVEL=INFO",
"PREFECT__GENERAL__SKIP_ALREADY_DONE_TASKS=false",
"PREFECT__BACKEND=server",
"PREFECT__CLOUD__API=<http://host.docker.internal:4200>",
"PREFECT__CLOUD__AUTH_TOKEN=",
"PREFECT__CLOUD__AGENT__LABELS=['prefect-development']",
"PREFECT__CONTEXT__FLOW_RUN_ID=275a48b7-9d79-439f-b0be-3fdac0f617a7",
"PREFECT__CONTEXT__FLOW_ID=0177a31d-a680-4734-aef4-ce683a97f35a",
"PREFECT__CONTEXT__IMAGE=cdwh/cdwh-flow:prefect-development",
"PREFECT__CLOUD__USE_LOCAL_SECRETS=false",
"PREFECT__LOGGING__LOG_TO_CLOUD=true",
"PREFECT__ENGINE__FLOW_RUNNER__DEFAULT_CLASS=prefect.engine.cloud.CloudFlowRunner",
"PREFECT__ENGINE__TASK_RUNNER__DEFAULT_CLASS=prefect.engine.cloud.CloudTaskRunner",
"PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"LANG=C.UTF-8",
"GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568",
"PYTHON_VERSION=3.8.10",
"PYTHON_PIP_VERSION=21.1.1",
"PYTHON_GET_PIP_URL=<https://github.com/pypa/get-pip/raw/1954f15b3f102ace496a34a013ea76b061535bd2/public/get-pip.py>",
"PYTHON_GET_PIP_SHA256=f499d76e0149a673fb8246d88e116db589afbd291739bd84f2cd9a7bca7b6993",
"LC_ALL=C.UTF-8",
"PREFECT__USER_CONFIG_PATH=./src/cdwhprefect/config.toml",
"REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt"
],
Zanie
05/21/2021, 4:50 PMv20.10.6
though so I am surprised it's failingdocker run -t alpine:latest /bin/sh -c "ping host.docker.internal"
?docker run --rm alpine nslookup host.docker.internal
)Michael Hadorn
05/21/2021, 5:28 PM> docker run -t alpine:latest /bin/sh -c "ping host.docker.internal"
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
540db60ca938: Pull complete
Digest: sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f
Status: Downloaded newer image for alpine:latest
ping: bad address 'host.docker.internal'
❯ docker run --rm alpine nslookup host.docker.internal
Server: 10.X.XX.X
Address: 10.X.XX.X:XX
** server can't find host.docker.internal: NXDOMAIN
** server can't find host.docker.internal: NXDOMAIN
This is the same on the prd system, where we didn't install the prefect upgrade.
Because it's working on 0.14.17. I think we have to dig in the python code.
But first i will try it also on a new linux system.Zanie
05/26/2021, 2:42 PMhost.docker.internal
as a work around for a bug in docker where it would not be consistently availableMichael Hadorn
05/26/2021, 3:05 PMZanie
05/26/2021, 3:15 PMmoby
link I provided there's an issue where people complain that host.docker.internal
is not consistently resolvable in their containers; this was something we patched by forcing it to be resolvable then removed because it appeared that the fix had been released in DockerMichael Hadorn
05/26/2021, 3:22 PMZanie
05/26/2021, 3:29 PMMichael Hadorn
05/26/2021, 3:31 PM