Thread
#prefect-server
    Raúl Mansilla

    Raúl Mansilla

    1 year ago
    Hello all, I´m trying to get a flow running using a docker agent that it´s already running in docker…I get the agent connected to prefect server and the flow gets to the agent when I run it, but It never finish…
    The error I´m getting via Agent Logs is:
    [2021-06-16 16:27:08,347] INFO - DockerTesting | Registering agent...
    [2021-06-16 16:27:08,366] INFO - DockerTesting | Registration successful!
    
     ____            __           _        _                    _
    |  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
    | |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
    |  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
    |_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|
                                               |___/
    
    [2021-06-16 16:27:08,371] INFO - DockerTesting | Starting DockerAgent with labels ['docker']
    [2021-06-16 16:27:08,371] INFO - DockerTesting | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
    [2021-06-16 16:27:08,371] INFO - DockerTesting | Waiting for flow runs...
    [2021-06-16 16:27:34,047] INFO - DockerTesting | Deploying flow run 1cec9a7b-25d2-44b0-8987-6f875c9a9bad to execution environment...
    [2021-06-16 16:27:34,097] INFO - DockerTesting | Pulling image rmansillasoto/prefect_docker_python_38:v1...
    [2021-06-16 16:27:34,977] INFO - DockerTesting | Successfully pulled image rmansillasoto/prefect_docker_python_38:v1
    [2021-06-16 16:27:35,316] INFO - DockerTesting | Completed deployment of flow run 1cec9a7b-25d2-44b0-8987-6f875c9a9bad
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 169, in _new_conn
        conn = connection.create_connection(
      File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 96, in create_connection
        raise err
      File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 86, in create_connection
        sock.connect(sa)
    ConnectionRefusedError: [Errno 111] Connection refused
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
        httplib_response = self._make_request(
      File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
        conn.request(method, url, **httplib_request_kw)
      File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 234, in request
        super(HTTPConnection, self).request(method, url, body=body, headers=headers)
      File "/usr/local/lib/python3.8/http/client.py", line 1252, in request
        self._send_request(method, url, body, headers, encode_chunked)
      File "/usr/local/lib/python3.8/http/client.py", line 1298, in _send_request
        self.endheaders(body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.8/http/client.py", line 1247, in endheaders
        self._send_output(message_body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.8/http/client.py", line 1007, in _send_output
        self.send(msg)
      File "/usr/local/lib/python3.8/http/client.py", line 947, in send
        self.connect()
      File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 200, in connect
        conn = self._new_conn()
      File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 181, in _new_conn
        raise NewConnectionError(
    urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fd972b16280>: Failed to establish a new connection: [Errno 111] Connection refused
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
        resp = conn.urlopen(
      File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
        retries = retries.increment(
      File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 574, in increment
        raise MaxRetryError(_pool, url, error or ResponseError(cause))
    urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='0.0.0.0', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd972b16280>: Failed to establish a new connection: [Errno 111] Connection refused'))
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/bin/prefect", line 8, in <module>
        sys.exit(cli())
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
        return self.main(*args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
        rv = self.invoke(ctx)
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
        return callback(*args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/prefect/cli/execute.py", line 49, in flow_run
        result = client.graphql(query)
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 298, in graphql
        result = <http://self.post|self.post>(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 213, in post
        response = self._request(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 459, in _request
        response = self._send_request(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 351, in _send_request
        response = <http://session.post|session.post>(
      File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 590, in post
        return self.request('POST', url, data=data, json=json, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
        resp = self.send(prep, **send_kwargs)
      File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
        r = adapter.send(request, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
        raise ConnectionError(e, request=request)
    requests.exceptions.ConnectionError: HTTPConnectionPool(host='0.0.0.0', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd972b16280>: Failed to establish a new connection: [Errno 111] Connection refused'))
    looks like I can´t connect again to port 4200
    The flow is:
    from prefect import task, Flow
    from prefect.run_configs import DockerRun
    from prefect.storage import CodeCommit
    import requests
    
    
    @task
    def overon_get_missing(days):
        url = "<https://xxxxxxxx.execute-api.eu-west-1.amazonaws.com/prod/api/v1/getMissing/>"+ str(days)
        response = <http://requests.post|requests.post>(url)
        missingList = response.json()
        return missingList
    
    run_config = DockerRun(
        env={'AWS_ACCESS_KEY_ID':'XXXXXXXXX',
        'AWS_SECRET_ACCESS_KEY':'XXXXXXXX',
        'AWS_DEFAULT_REGION':'eu-west-1'},
        image='rmansillasoto/prefect_docker_python_38:v1'
    )
    
    #flow = Flow("docker-flow",storage=CodeCommit(repo="prefect_flows", path="flows/test_docker.py", commit = "master"),
    #            run_config=run_config)
    
    with Flow("docker-flow") as flow:
        overon_get_missing(30)
    flow.storage = CodeCommit(repo="prefect_flows", path="flows/test_docker.py", commit="master")
    flow.run_config = run_config
    flow.register(project_name='raulTesting', labels=['docker'], idempotency_key=flow.serialized_hash())
    
    #flow.run(days=30)
    and the command I use to launch the agent is:
    docker run -it --rm --net=host --name dockerAgentPrefectDev -v /var/run/docker.sock:/var/run/docker.sock  rmansillasoto/prefect_docker_agent:v1 /usr/local/bin/prefect agent docker start --name DockerTesting --label docker --base-url <unix://var/run/docker.sock> --api <http://0.0.0.0:4200> --no-docker-interface --show-flow-logs
    FYI if I run the agent outside docker, directly on the prefect server…the flow works
    m

    Mariia Kerimova

    1 year ago
    Could you change the ip value you passing to --api IP for your server and run it again?
    Actually my bad, it should not be the issue, let me try to reproduce, and I'll get back to you
    Just to double check I understand the issue correctly. Instead of running
    prefect agent docker start
    , you want to use your custom container based on this image
    rmansillasoto/prefect_docker_agent:v1
    , but flow run fails with connection refused error, right?
    Raúl Mansilla

    Raúl Mansilla

    1 year ago
    hello! i would like to run this command inside the docker container…it is a custom prefect container
    I mean, I would like to have the docker agent running inside the docker container.
    m

    Mariia Kerimova

    1 year ago
    Could you please test following command and let me know if you can see the agent is running and you see it in UI:
    docker run -it --rm --net=host -v /var/run/docker.sock:/var/run/docker.sock prefecthq/prefect:latest /usr/local/bin/prefect backend server && prefect agent docker start --name DockerTesting --label docker --base-url <unix://var/run/docker.sock> --api http://<IP of your server>:4200
    Raúl Mansilla

    Raúl Mansilla

    1 year ago
    With my command the agent runs and appear in UI as well, but the jobs never get done
    and with your command, happens the same
    the job remains in submitted for execution…
    ____            __           _        _                    _
    |  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
    | |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
    |  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
    |_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|
                                               |___/
    
    [2021-06-16 20:39:23,990] INFO - DockerTesting | Starting DockerAgent with labels ['docker']
    [2021-06-16 20:39:23,990] INFO - DockerTesting | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
    [2021-06-16 20:39:23,990] INFO - DockerTesting | Agent connecting to the Prefect API at <http://0.0.0.0:4200>
    [2021-06-16 20:39:23,996] INFO - DockerTesting | Waiting for flow runs...
    [2021-06-16 20:39:49,670] INFO - DockerTesting | Found 1 flow run(s) to submit for execution.
    [2021-06-16 20:39:49,713] INFO - DockerTesting | Deploying flow run 77ec2339-00c8-4ff1-9bac-ac8510a07325
    [2021-06-16 20:39:49,714] INFO - DockerTesting | Pulling image rmansillasoto/prefect_docker_python_38:v1...
    [2021-06-16 20:39:50,645] INFO - DockerTesting | Successfully pulled image rmansillasoto/prefect_docker_python_38:v1...
    is there any limit of connections to port 4200 on prefect server?
    m

    Mariia Kerimova

    1 year ago
    I see, thank you for more details. I'll continue looking into this issue. There is no limit on prefect server connections.
    Raúl Mansilla

    Raúl Mansilla

    1 year ago
    ok, I´m trying to register the flow from my laptop instead of doing locally in the remote prefect server…but I get also a connection problem with port 4200…rllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='http', port=80): Max retries exceeded with url: <//xxxxxx.xxx-tech.net/:4200/graphql> (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff70462c760>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))
    m

    Mariia Kerimova

    1 year ago
    it looks like it's trying to use port 80, instead of 4200
    Raúl Mansilla

    Raúl Mansilla

    1 year ago
    now it worked…there was an extra / in the url 😕
    m

    Mariia Kerimova

    1 year ago
    haha, what a great fix, I'm glad you figured it out 👍
    Raúl Mansilla

    Raúl Mansilla

    1 year ago
    I have advanced a little bit and now my container running the docker agent inside raises this error:
    requests.exceptions.ConnectionError: HTTPConnectionPool(host='host.docker.internal', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd427d7d280>: Failed to establish a new connection: [Errno -2] Name or service not known'))
    I think here we can find some info: https://github.com/PrefectHQ/server/issues/25
    @Mariia Kerimova hello! any update on this?
    m

    Mariia Kerimova

    1 year ago
    Hello Raul! Sorry for delay, I was a bit interrupted today. I'm going to test it tonight, and I'll update you asap.
    Hello Raul! Can you provide the info about the environment you are using and versions (docker docker-compose)?
    Raúl Mansilla

    Raúl Mansilla

    1 year ago
    I will check it in a few hours
    @Mariia Kerimova here you have:
    ubuntu@ip-10-10-5-107:~$ docker version
    Client: Docker Engine - Community
     Version:           20.10.6
     API version:       1.41
     Go version:        go1.13.15
     Git commit:        370c289
     Built:             Fri Apr  9 22:47:17 2021
     OS/Arch:           linux/amd64
     Context:           default
     Experimental:      true
    
    Server: Docker Engine - Community
     Engine:
      Version:          20.10.6
      API version:      1.41 (minimum version 1.12)
      Go version:       go1.13.15
      Git commit:       8728dd2
      Built:            Fri Apr  9 22:45:28 2021
      OS/Arch:          linux/amd64
      Experimental:     false
     containerd:
      Version:          1.4.4
      GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
     runc:
      Version:          1.0.0-rc93
      GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
     docker-init:
      Version:          0.19.0
      GitCommit:        de40ad0
    ubuntu@ip-10-10-5-107:~$ docker-compose version
    docker-compose version 1.27.4, build 40524192
    docker-py version: 4.3.1
    CPython version: 3.7.7
    OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019
    its an ubuntu 20.04.2 ec2 with Prefect 0.14.19