Hello all I m trying to get a flow running using a docker ag Prefect Community #prefect-server

Hello all, I´m trying to get a flow running using ...

Raúl Mansilla

06/16/2021, 4:32 PM

Hello all, I´m trying to get a flow running using a docker agent that it´s already running in docker…I get the agent connected to prefect server and the flow gets to the agent when I run it, but It never finish…

Raúl Mansilla

06/16/2021, 4:34 PM

The error I´m getting via Agent Logs is:

Copy code

[2021-06-16 16:27:08,347] INFO - DockerTesting | Registering agent...
[2021-06-16 16:27:08,366] INFO - DockerTesting | Registration successful!

 ____            __           _        _                    _
|  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
| |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
|  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
|_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|
                                           |___/

[2021-06-16 16:27:08,371] INFO - DockerTesting | Starting DockerAgent with labels ['docker']
[2021-06-16 16:27:08,371] INFO - DockerTesting | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2021-06-16 16:27:08,371] INFO - DockerTesting | Waiting for flow runs...
[2021-06-16 16:27:34,047] INFO - DockerTesting | Deploying flow run 1cec9a7b-25d2-44b0-8987-6f875c9a9bad to execution environment...
[2021-06-16 16:27:34,097] INFO - DockerTesting | Pulling image rmansillasoto/prefect_docker_python_38:v1...
[2021-06-16 16:27:34,977] INFO - DockerTesting | Successfully pulled image rmansillasoto/prefect_docker_python_38:v1
[2021-06-16 16:27:35,316] INFO - DockerTesting | Completed deployment of flow run 1cec9a7b-25d2-44b0-8987-6f875c9a9bad
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.8/http/client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1007, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 947, in send
    self.connect()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 200, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 181, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fd972b16280>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='0.0.0.0', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd972b16280>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/prefect", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/cli/execute.py", line 49, in flow_run
    result = client.graphql(query)
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 298, in graphql
    result = <http://self.post|self.post>(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 213, in post
    response = self._request(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 459, in _request
    response = self._send_request(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 351, in _send_request
    response = <http://session.post|session.post>(
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 590, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='0.0.0.0', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd972b16280>: Failed to establish a new connection: [Errno 111] Connection refused'))

Raúl Mansilla

06/16/2021, 4:36 PM

looks like I can´t connect again to port 4200

Raúl Mansilla

06/16/2021, 4:38 PM

The flow is:

Copy code

from prefect import task, Flow
from prefect.run_configs import DockerRun
from prefect.storage import CodeCommit
import requests


@task
def overon_get_missing(days):
    url = "<https://xxxxxxxx.execute-api.eu-west-1.amazonaws.com/prod/api/v1/getMissing/>"+ str(days)
    response = <http://requests.post|requests.post>(url)
    missingList = response.json()
    return missingList

run_config = DockerRun(
    env={'AWS_ACCESS_KEY_ID':'XXXXXXXXX',
    'AWS_SECRET_ACCESS_KEY':'XXXXXXXX',
    'AWS_DEFAULT_REGION':'eu-west-1'},
    image='rmansillasoto/prefect_docker_python_38:v1'
)

#flow = Flow("docker-flow",storage=CodeCommit(repo="prefect_flows", path="flows/test_docker.py", commit = "master"),
#            run_config=run_config)

with Flow("docker-flow") as flow:
    overon_get_missing(30)
flow.storage = CodeCommit(repo="prefect_flows", path="flows/test_docker.py", commit="master")
flow.run_config = run_config
flow.register(project_name='raulTesting', labels=['docker'], idempotency_key=flow.serialized_hash())

#flow.run(days=30)

Raúl Mansilla

06/16/2021, 4:40 PM

and the command I use to launch the agent is:

docker run -it --rm --net=host --name dockerAgentPrefectDev -v /var/run/docker.sock:/var/run/docker.sock  rmansillasoto/prefect_docker_agent:v1 /usr/local/bin/prefect agent docker start --name DockerTesting --label docker --base-url <unix://var/run/docker.sock> --api <http://0.0.0.0:4200> --no-docker-interface --show-flow-logs

Raúl Mansilla

06/16/2021, 5:09 PM

FYI if I run the agent outside docker, directly on the prefect server…the flow works

Mariia Kerimova

06/16/2021, 6:27 PM

Could you change the ip value you passing to --api IP for your server and run it again?

Mariia Kerimova

06/16/2021, 6:35 PM

Actually my bad, it should not be the issue, let me try to reproduce, and I'll get back to you

Mariia Kerimova

06/16/2021, 6:50 PM

Just to double check I understand the issue correctly. Instead of running

prefect agent docker start

, you want to use your custom container based on this image

rmansillasoto/prefect_docker_agent:v1

, but flow run fails with connection refused error, right?

Raúl Mansilla

06/16/2021, 7:53 PM

hello! i would like to run this command inside the docker container…it is a custom prefect container

Raúl Mansilla

06/16/2021, 7:56 PM

I mean, I would like to have the docker agent running inside the docker container.

Mariia Kerimova

06/16/2021, 8:30 PM

Could you please test following command and let me know if you can see the agent is running and you see it in UI:

Copy code

docker run -it --rm --net=host -v /var/run/docker.sock:/var/run/docker.sock prefecthq/prefect:latest /usr/local/bin/prefect backend server && prefect agent docker start --name DockerTesting --label docker --base-url <unix://var/run/docker.sock> --api http://<IP of your server>:4200

Raúl Mansilla

06/16/2021, 8:38 PM

With my command the agent runs and appear in UI as well, but the jobs never get done

Raúl Mansilla

06/16/2021, 8:40 PM

and with your command, happens the same

Raúl Mansilla

06/16/2021, 8:41 PM

the job remains in submitted for execution…

Copy code

____            __           _        _                    _
|  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
| |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
|  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
|_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|
                                           |___/

[2021-06-16 20:39:23,990] INFO - DockerTesting | Starting DockerAgent with labels ['docker']
[2021-06-16 20:39:23,990] INFO - DockerTesting | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2021-06-16 20:39:23,990] INFO - DockerTesting | Agent connecting to the Prefect API at <http://0.0.0.0:4200>
[2021-06-16 20:39:23,996] INFO - DockerTesting | Waiting for flow runs...
[2021-06-16 20:39:49,670] INFO - DockerTesting | Found 1 flow run(s) to submit for execution.
[2021-06-16 20:39:49,713] INFO - DockerTesting | Deploying flow run 77ec2339-00c8-4ff1-9bac-ac8510a07325
[2021-06-16 20:39:49,714] INFO - DockerTesting | Pulling image rmansillasoto/prefect_docker_python_38:v1...
[2021-06-16 20:39:50,645] INFO - DockerTesting | Successfully pulled image rmansillasoto/prefect_docker_python_38:v1...

Raúl Mansilla

06/16/2021, 8:50 PM

is there any limit of connections to port 4200 on prefect server?

Mariia Kerimova

06/16/2021, 8:51 PM

I see, thank you for more details. I'll continue looking into this issue. There is no limit on prefect server connections.

Raúl Mansilla

06/16/2021, 8:54 PM

ok, I´m trying to register the flow from my laptop instead of doing locally in the remote prefect server…but I get also a connection problem with port 4200…`rllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='http', port=80): Max retries exceeded with url: <//xxxxxx.xxx-tech.net/:4200/graphql> (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff70462c760>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))`

Mariia Kerimova

06/16/2021, 8:56 PM

it looks like it's trying to use port 80, instead of 4200

Raúl Mansilla

06/16/2021, 9:01 PM

now it worked…there was an extra / in the url :-/

Mariia Kerimova

06/16/2021, 9:02 PM

haha, what a great fix, I'm glad you figured it out 👍

Raúl Mansilla

06/16/2021, 9:35 PM

I have advanced a little bit and now my container running the docker agent inside raises this error:

requests.exceptions.ConnectionError: HTTPConnectionPool(host='host.docker.internal', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd427d7d280>: Failed to establish a new connection: [Errno -2] Name or service not known'))

Raúl Mansilla

06/16/2021, 9:39 PM

I think here we can find some info: https://github.com/PrefectHQ/server/issues/25

👀 1

Raúl Mansilla

06/16/2021, 10:07 PM

and also… https://prefect-community.slack.com/archives/C014Z8DPDSR/p1621438554048200

👀 1

Raúl Mansilla

06/16/2021, 10:09 PM

and https://github.com/PrefectHQ/prefect/issues/2324

Raúl Mansilla

06/17/2021, 8:24 PM

@Mariia Kerimova hello! any update on this?

Mariia Kerimova

06/17/2021, 9:31 PM

Hello Raul! Sorry for delay, I was a bit interrupted today. I'm going to test it tonight, and I'll update you asap.

Mariia Kerimova

06/18/2021, 3:36 PM

Hello Raul! Can you provide the info about the environment you are using and versions (docker docker-compose)?

Raúl Mansilla

06/21/2021, 5:15 PM

I will check it in a few hours

Raúl Mansilla

06/21/2021, 10:27 PM

@Mariia Kerimova here you have:

Copy code

ubuntu@ip-10-10-5-107:~$ docker version
Client: Docker Engine - Community
 Version:           20.10.6
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        370c289
 Built:             Fri Apr  9 22:47:17 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.6
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8728dd2
  Built:            Fri Apr  9 22:45:28 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Raúl Mansilla

06/21/2021, 10:27 PM

Copy code

ubuntu@ip-10-10-5-107:~$ docker-compose version
docker-compose version 1.27.4, build 40524192
docker-py version: 4.3.1
CPython version: 3.7.7
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019

Raúl Mansilla

06/22/2021, 7:41 AM

its an ubuntu 20.04.2 ec2 with Prefect 0.14.19

Open in Slack

Previous Next