Hello all I m trying to get a flow running using a docker ag

Question

Hello all I m trying to get a flow running using a docker agent that it s already running in docker I get the agent connected to prefect server and the flow gets to the agent when I run it but It never finish

Raúl Mansilla · Answer

The error I m getting via Agent Logs is ``` 2021 06 16 16 27 08 347 INFO DockerTesting | Registering agent 2021 06 16 16 27 08 366 INFO DockerTesting | Registration successful | | | | | | | | | | | | ` | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 2021 06 16 16 27 08 371 INFO DockerTesting | Starting DockerAgent with labels docker 2021 06 16 16 27 08 371 INFO DockerTesting | Agent documentation can be found at 2021 06 16 16 27 08 371 INFO DockerTesting | Waiting for flow runs 2021 06 16 16 27 34 047 INFO DockerTesting | Deploying flow run 1cec9a7b 25d2 44b0 8987 6f875c9a9bad to execution environment 2021 06 16 16 27 34 097 INFO DockerTesting | Pulling image rmansillasoto prefect docker python 38 v1 2021 06 16 16 27 34 977 INFO DockerTesting | Successfully pulled image rmansillasoto prefect docker python 38 v1 2021 06 16 16 27 35 316 INFO DockerTesting | Completed deployment of flow run 1cec9a7b 25d2 44b0 8987 6f875c9a9bad Traceback most recent call last File usr local lib python3 8 site packages urllib3 connection py line 169 in new conn conn = connection create connection File usr local lib python3 8 site packages urllib3 util connection py line 96 in create connection raise err File usr local lib python3 8 site packages urllib3 util connection py line 86 in create connection sock connect sa ConnectionRefusedError Errno 111 Connection refused During handling of the above exception another exception occurred Traceback most recent call last File usr local lib python3 8 site packages urllib3 connectionpool py line 699 in urlopen httplib response = self make request File usr local lib python3 8 site packages urllib3 connectionpool py line 394 in make request conn request method url httplib request kw File usr local lib python3 8 site packages urllib3 connection py line 234 in request super HTTPConnection self request method url body=body headers=headers File usr local lib python3 8 http client py line 1252 in request self send request method url body headers encode chunked File usr local lib python3 8 http client py line 1298 in send request self endheaders body encode chunked=encode chunked File usr local lib python3 8 http client py line 1247 in endheaders self send output message body encode chunked=encode chunked File usr local lib python3 8 http client py line 1007 in send output self send msg File usr local lib python3 8 http client py line 947 in send self connect File usr local lib python3 8 site packages urllib3 connection py line 200 in connect conn = self new conn File usr local lib python3 8 site packages urllib3 connection py line 181 in new conn raise NewConnectionError urllib3 exceptions NewConnectionError lt urllib3 connection HTTPConnection object at 0x7fd972b16280 gt Failed to establish a new connection Errno 111 Connection refused During handling of the above exception another exception occurred Traceback most recent call last File usr local lib python3 8 site packages requests adapters py line 439 in send resp = conn urlopen File usr local lib python3 8 site packages urllib3 connectionpool py line 755 in urlopen retries = retries increment File usr local lib python3 8 site packages urllib3 util retry py line 574 in increment raise MaxRetryError pool url error or ResponseError cause urllib3 exceptions MaxRetryError HTTPConnectionPool host= 0 0 0 0 port=4200 Max retries exceeded with url graphql Caused by NewConnectionError lt urllib3 connection HTTPConnection object at 0x7fd972b16280 gt Failed to establish a new connection Errno 111 Connection refused During handling of the above exception another exception occurred Traceback most recent call last File usr local bin prefect line 8 in lt module gt sys exit cli File usr local lib python3 8 site packages click core py line 829 in call return self main args kwargs File usr local lib python3 8 site packages click core py line 782 in main rv = self invoke ctx File usr local lib python3 8 site packages click core py line 1259 in invoke return process result sub ctx command invoke sub ctx File usr local lib python3 8 site packages click core py line 1259 in invoke return process result sub ctx command invoke sub ctx File usr local lib python3 8 site packages click core py line 1066 in invoke return ctx invoke self callback ctx params File usr local lib python3 8 site packages click core py line 610 in invoke return callback args kwargs File usr local lib python3 8 site packages prefect cli execute py line 49 in flow run result = client graphql query File usr local lib python3 8 site packages prefect client client py line 298 in graphql result = File usr local lib python3 8 site packages prefect client client py line 213 in post response = self request File usr local lib python3 8 site packages prefect client client py line 459 in request response = self send request File usr local lib python3 8 site packages prefect client client py line 351 in send request response = File usr local lib python3 8 site packages requests sessions py line 590 in post return self request POST url data=data json=json kwargs File usr local lib python3 8 site packages requests sessions py line 542 in request resp = self send prep send kwargs File usr local lib python3 8 site packages requests sessions py line 655 in send r = adapter send request kwargs File usr local lib python3 8 site packages requests adapters py line 516 in send raise ConnectionError e request=request requests exceptions ConnectionError HTTPConnectionPool host= 0 0 0 0 port=4200 Max retries exceeded with url graphql Caused by NewConnectionError lt urllib3 connection HTTPConnection object at 0x7fd972b16280 gt Failed to establish a new connection Errno 111 Connection refused ```

Raúl Mansilla · Answer

looks like I can t connect again to port 4200

Raúl Mansilla · Answer

The flow is ```from prefect import task Flow from prefect run configs import DockerRun from prefect storage import CodeCommit import requests task def overon get missing days url = + str days response = url missingList = response json return missingList run config = DockerRun env= AWS ACCESS KEY ID XXXXXXXXX AWS SECRET ACCESS KEY XXXXXXXX AWS DEFAULT REGION eu west 1 image= rmansillasoto prefect docker python 38 v1 flow = Flow docker flow storage=CodeCommit repo= prefect flows path= flows test docker py commit = master run config=run config with Flow docker flow as flow overon get missing 30 flow storage = CodeCommit repo= prefect flows path= flows test docker py commit= master flow run config = run config flow register project name= raulTesting labels= docker idempotency key=flow serialized hash flow run days=30 ```

Raúl Mansilla · Answer

and the command I use to launch the agent is `docker run it rm net=host name dockerAgentPrefectDev v var run docker sock var run docker sock rmansillasoto prefect docker agent v1 usr local bin prefect agent docker start name DockerTesting label docker base url api no docker interface show flow logs`

Raúl Mansilla · Answer

FYI if I run the agent outside docker directly on the prefect server the flow works

Mariia Kerimova · Answer

Could you change the ip value you passing to api IP for your server and run it again

Mariia Kerimova · Answer

Actually my bad it should not be the issue let me try to reproduce and I ll get back to you

Mariia Kerimova · Answer

Just to double check I understand the issue correctly Instead of running `prefect agent docker start` you want to use your custom container based on this image `rmansillasoto prefect docker agent v1` but flow run fails with connection refused error right

Raúl Mansilla · Answer

hello i would like to run this command inside the docker container it is a custom prefect container

Raúl Mansilla · Answer

I mean I would like to have the docker agent running inside the docker container

Mariia Kerimova · Answer

Could you please test following command and let me know if you can see the agent is running and you see it in UI ```docker run it rm net=host v var run docker sock var run docker sock prefecthq prefect latest usr local bin prefect backend server amp amp prefect agent docker start name DockerTesting label docker base url <unix var run docker sock> api http lt IP of your server gt 4200```

Raúl Mansilla · Answer

With my command the agent runs and appear in UI as well but the jobs never get done

Raúl Mansilla · Answer

and with your command happens the same

Raúl Mansilla · Answer

the job remains in submitted for execution ``` | | | | | | | | | | | | ` | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 2021 06 16 20 39 23 990 INFO DockerTesting | Starting DockerAgent with labels docker 2021 06 16 20 39 23 990 INFO DockerTesting | Agent documentation can be found at 2021 06 16 20 39 23 990 INFO DockerTesting | Agent connecting to the Prefect API at 2021 06 16 20 39 23 996 INFO DockerTesting | Waiting for flow runs 2021 06 16 20 39 49 670 INFO DockerTesting | Found 1 flow run s to submit for execution 2021 06 16 20 39 49 713 INFO DockerTesting | Deploying flow run 77ec2339 00c8 4ff1 9bac ac8510a07325 2021 06 16 20 39 49 714 INFO DockerTesting | Pulling image rmansillasoto prefect docker python 38 v1 2021 06 16 20 39 50 645 INFO DockerTesting | Successfully pulled image rmansillasoto prefect docker python 38 v1 ```

Raúl Mansilla · Answer

is there any limit of connections to port 4200 on prefect server

Mariia Kerimova · Answer

I see thank you for more details I ll continue looking into this issue There is no limit on prefect server connections

Raúl Mansilla · Answer

ok I m trying to register the flow from my laptop instead of doing locally in the remote prefect server but I get also a connection problem with port 4200 `rllib3 exceptions MaxRetryError HTTPConnectionPool host= http port=80 Max retries exceeded with url lt xxxxxx xxx tech net 4200 graphql gt Caused by NewConnectionError lt urllib3 connection HTTPConnection object at 0x7ff70462c760 gt Failed to establish a new connection Errno 8 nodename nor servname provided or not known `

Mariia Kerimova · Answer

it looks like it s trying to use port 80 instead of 4200

Raúl Mansilla · Answer

now it worked there was an extra in the url

Mariia Kerimova · Answer

haha what a great fix I m glad you figured it out +1

Raúl Mansilla · Answer

I have advanced a little bit and now my container running the docker agent inside raises this error `requests exceptions ConnectionError HTTPConnectionPool host= host docker internal port=4200 Max retries exceeded with url graphql Caused by NewConnectionError lt urllib3 connection HTTPConnection object at 0x7fd427d7d280 gt Failed to establish a new connection Errno 2 Name or service not known `

Raúl Mansilla · Answer

I think here we can find some info <https github com PrefectHQ server issues 25>

Raúl Mansilla · Answer

and also <https prefect community slack com archives C014Z8DPDSR p1621438554048200>

Raúl Mansilla · Answer

and <https github com PrefectHQ prefect issues 2324>

Raúl Mansilla · Answer

< Mariia Kerimova> hello any update on this

Mariia Kerimova · Answer

Hello Raul Sorry for delay I was a bit interrupted today I m going to test it tonight and I ll update you asap

Mariia Kerimova · Answer

Hello Raul Can you provide the info about the environment you are using and versions docker docker compose

Raúl Mansilla · Answer

I will check it in a few hours

Raúl Mansilla · Answer

< Mariia Kerimova> here you have ```ubuntu ip 10 10 5 107 ~$ docker version Client Docker Engine Community Version 20 10 6 API version 1 41 Go version go1 13 15 Git commit 370c289 Built Fri Apr 9 22 47 17 2021 OS Arch linux amd64 Context default Experimental true Server Docker Engine Community Engine Version 20 10 6 API version 1 41 minimum version 1 12 Go version go1 13 15 Git commit 8728dd2 Built Fri Apr 9 22 45 28 2021 OS Arch linux amd64 Experimental false containerd Version 1 4 4 GitCommit 05f951a3781f4f2c1911b05e61c160e9c30eaa8e runc Version 1 0 0 rc93 GitCommit 12644e614e25b05da6fd08a38ffa0cfe1903fdec docker init Version 0 19 0 GitCommit de40ad0```

Raúl Mansilla · Answer

```ubuntu ip 10 10 5 107 ~$ docker compose version docker compose version 1 27 4 build 40524192 docker py version 4 3 1 CPython version 3 7 7 OpenSSL version OpenSSL 1 1 0l 10 Sep 2019```

Raúl Mansilla · Answer

its an ubuntu 20 04 2 ec2 with Prefect 0 14 19