I am trying to run a Prefect flow in a docker worker pool ov Prefect Community #prefect-docker

I am trying to run a Prefect flow in a docker work...

Amruth VVKP

01/24/2024, 7:38 AM

I am trying to run a Prefect flow in a docker worker pool over a self hosted Prefect server. My

docker-compose.yml

is starting up a docker type Prefect work pool which is resulting in this error -

Copy code

2024-01-24 07:35:32 07:35:32.358 | ERROR   | prefect.flow_runs.worker - Failed to submit flow run '7639a755-0d52-4d45-874d-ca7aa17e5f81' to infrastructure.
2024-01-24 07:35:32 Traceback (most recent call last):
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
2024-01-24 07:35:32     httplib_response = self._make_request(
2024-01-24 07:35:32                        ^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 416, in _make_request
2024-01-24 07:35:32     conn.request(method, url, **httplib_request_kw)
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 244, in request
2024-01-24 07:35:32     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/http/client.py", line 1294, in request
2024-01-24 07:35:32     self._send_request(method, url, body, headers, encode_chunked)
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/http/client.py", line 1340, in _send_request
2024-01-24 07:35:32     self.endheaders(body, encode_chunked=encode_chunked)
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/http/client.py", line 1289, in endheaders
2024-01-24 07:35:32     self._send_output(message_body, encode_chunked=encode_chunked)
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/http/client.py", line 1048, in _send_output
2024-01-24 07:35:32     self.send(msg)
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/http/client.py", line 986, in send
2024-01-24 07:35:32     self.connect()
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/docker/transport/unixconn.py", line 27, in connect
2024-01-24 07:35:32     sock.connect(self.unix_socket)
2024-01-24 07:35:32 FileNotFoundError: [Errno 2] No such file or directory
2024-01-24 07:35:32 
2024-01-24 07:35:32 During handling of the above exception, another exception occurred:
2024-01-24 07:35:32 
2024-01-24 07:35:32 Traceback (most recent call last):
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
2024-01-24 07:35:32     resp = conn.urlopen(
2024-01-24 07:35:32            ^^^^^^^^^^^^^
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 799, in urlopen
2024-01-24 07:35:32     retries = retries.increment(
2024-01-24 07:35:32               ^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 550, in increment
2024-01-24 07:35:32     raise six.reraise(type(error), error, _stacktrace)
2024-01-24 07:35:32           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/urllib3/packages/six.py", line 769, in reraise
2024-01-24 07:35:32     raise value.with_traceback(tb)
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
2024-01-24 07:35:32     httplib_response = self._make_request(
2024-01-24 07:35:32                        ^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 416, in _make_request
2024-01-24 07:35:32     conn.request(method, url, **httplib_request_kw)
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 244, in request
2024-01-24 07:35:32     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/http/client.py", line 1294, in request
2024-01-24 07:35:32     self._send_request(method, url, body, headers, encode_chunked)
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/http/client.py", line 1340, in _send_request
2024-01-24 07:35:32     self.endheaders(body, encode_chunked=encode_chunked)
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/http/client.py", line 1289, in endheaders
2024-01-24 07:35:32     self._send_output(message_body, encode_chunked=encode_chunked)
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/http/client.py", line 1048, in _send_output
2024-01-24 07:35:32     self.send(msg)
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/http/client.py", line 986, in send
2024-01-24 07:35:32     self.connect()
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/docker/transport/unixconn.py", line 27, in connect
2024-01-24 07:35:32     sock.connect(self.unix_socket)
2024-01-24 07:35:32 urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
2024-01-24 07:35:32 
2024-01-24 07:35:32 During handling of the above exception, another exception occurred:
2024-01-24 07:35:32 
2024-01-24 07:35:32 Traceback (most recent call last):
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
2024-01-24 07:35:32     return self.version(api_version=False)["ApiVersion"]
2024-01-24 07:35:32            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/docker/api/daemon.py", line 181, in version
2024-01-24 07:35:32     return self._result(self._get(url), json=True)
2024-01-24 07:35:32                         ^^^^^^^^^^^^^^
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/docker/utils/decorators.py", line 46, in inner
2024-01-24 07:35:32     return f(self, *args, **kwargs)
2024-01-24 07:35:32            ^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 237, in _get
2024-01-24 07:35:32     return self.get(url, **self._set_request_timeout(kwargs))
2024-01-24 07:35:32            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
2024-01-24 07:35:32     return self.request("GET", url, **kwargs)
2024-01-24 07:35:32            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
2024-01-24 07:35:32     resp = self.send(prep, **send_kwargs)
2024-01-24 07:35:32            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
2024-01-24 07:35:32     r = adapter.send(request, **kwargs)
2024-01-24 07:35:32         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 501, in send
2024-01-24 07:35:32     raise ConnectionError(err, request=request)
2024-01-24 07:35:32 requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
2024-01-24 07:35:32 
2024-01-24 07:35:32 During handling of the above exception, another exception occurred:
2024-01-24 07:35:32 
2024-01-24 07:35:32 Traceback (most recent call last):
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/prefect_docker/worker.py", line 498, in _get_client
2024-01-24 07:35:32     docker_client = docker.from_env()
2024-01-24 07:35:32                     ^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/docker/client.py", line 96, in from_env
2024-01-24 07:35:32     return cls(
2024-01-24 07:35:32            ^^^^
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/docker/client.py", line 45, in __init__
2024-01-24 07:35:32     self.api = APIClient(*args, **kwargs)
2024-01-24 07:35:32                ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 197, in __init__
2024-01-24 07:35:32     self._version = self._retrieve_server_version()
2024-01-24 07:35:32                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
2024-01-24 07:35:32     raise DockerException(
2024-01-24 07:35:32 docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
2024-01-24 07:35:32 
2024-01-24 07:35:32 The above exception was the direct cause of the following exception:
2024-01-24 07:35:32 
2024-01-24 07:35:32 Traceback (most recent call last):
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/prefect/workers/base.py", line 896, in _submit_run_and_capture_errors
2024-01-24 07:35:32     result = await self.run(
2024-01-24 07:35:32              ^^^^^^^^^^^^^^^
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/prefect_docker/worker.py", line 417, in run
2024-01-24 07:35:32     container, created_event = await run_sync_in_worker_thread(
2024-01-24 07:35:32                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
2024-01-24 07:35:32     return await anyio.to_thread.run_sync(
2024-01-24 07:35:32            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/anyio/to_thread.py", line 33, in run_sync
2024-01-24 07:35:32     return await get_asynclib().run_sync_in_worker_thread(
2024-01-24 07:35:32            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
2024-01-24 07:35:32     return await future
2024-01-24 07:35:32            ^^^^^^^^^^^^
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 807, in run
2024-01-24 07:35:32     result = context.run(func, *args)
2024-01-24 07:35:32              ^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/prefect_docker/worker.py", line 548, in _create_and_start_container
2024-01-24 07:35:32     docker_client = self._get_client()
2024-01-24 07:35:32                     ^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32   File "/usr/local/lib/python3.11/site-packages/prefect_docker/worker.py", line 501, in _get_client
2024-01-24 07:35:32     raise RuntimeError("Could not connect to Docker.") from exc
2024-01-24 07:35:32 RuntimeError: Could not connect to Docker.
2024-01-24 07:35:32 07:35:32.364 | INFO    | prefect.flow_runs.worker - Completed submission of flow run '7639a755-0d52-4d45-874d-ca7aa17e5f81'
2024-01-24 07:35:32 07:35:32.388 | INFO    | prefect.flow_runs.worker - Reported flow run '7639a755-0d52-4d45-874d-ca7aa17e5f81' as crashed: Flow run could not be submitted to infrastructure

Amruth VVKP

01/24/2024, 7:42 AM

I've also tried to start the docker work pool outside the docker container assuming that the container doesn't have docker-in-docker accessibility. This is resulting in another error -

Copy code

> prefect worker start --pool 'misc-tasks' --type docker
Worker 'DockerWorker 0cf14eb9-301e-4e56-a9fd-eab160bd9b93' started!
07:41:31.372 | INFO    | prefect.flow_runs.worker - Worker 'DockerWorker 0cf14eb9-301e-4e56-a9fd-eab160bd9b93' submitting flow run '9d6ac137-9a6a-454d-b536-8c9d7600f163'
07:41:31.571 | ERROR   | prefect.flow_runs.worker - Failed to submit flow run '9d6ac137-9a6a-454d-b536-8c9d7600f163' to infrastructure.
Traceback (most recent call last):
  File "/home/amruthvvkp/.cache/pypoetry/virtualenvs/aws-client-xb2bGS7e-py3.11/lib/python3.11/site-packages/prefect/workers/base.py", line 896, in _submit_run_and_capture_errors
    result = await self.run(
             ^^^^^^^^^^^^^^^
  File "/home/amruthvvkp/.cache/pypoetry/virtualenvs/aws-client-xb2bGS7e-py3.11/lib/python3.11/site-packages/prefect_docker/worker.py", line 407, in run
    container, created_event = await run_sync_in_worker_thread(
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/amruthvvkp/.cache/pypoetry/virtualenvs/aws-client-xb2bGS7e-py3.11/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/amruthvvkp/.cache/pypoetry/virtualenvs/aws-client-xb2bGS7e-py3.11/lib/python3.11/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/amruthvvkp/.cache/pypoetry/virtualenvs/aws-client-xb2bGS7e-py3.11/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/home/amruthvvkp/.cache/pypoetry/virtualenvs/aws-client-xb2bGS7e-py3.11/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/amruthvvkp/.cache/pypoetry/virtualenvs/aws-client-xb2bGS7e-py3.11/lib/python3.11/site-packages/prefect_docker/worker.py", line 539, in _create_and_start_container
    container_settings = self._build_container_settings(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/amruthvvkp/.cache/pypoetry/virtualenvs/aws-client-xb2bGS7e-py3.11/lib/python3.11/site-packages/prefect_docker/worker.py", line 526, in _build_container_settings
    extra_hosts=configuration.get_extra_hosts(docker_client),
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/amruthvvkp/.cache/pypoetry/virtualenvs/aws-client-xb2bGS7e-py3.11/lib/python3.11/site-packages/prefect_docker/worker.py", line 315, in get_extra_hosts
    user_version = packaging.version.parse(
                   ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/amruthvvkp/.cache/pypoetry/virtualenvs/aws-client-xb2bGS7e-py3.11/lib/python3.11/site-packages/packaging/version.py", line 54, in parse
    return Version(version)
           ^^^^^^^^^^^^^^^^
  File "/home/amruthvvkp/.cache/pypoetry/virtualenvs/aws-client-xb2bGS7e-py3.11/lib/python3.11/site-packages/packaging/version.py", line 200, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
packaging.version.InvalidVersion: Invalid version: 'master'
07:41:31.590 | INFO    | prefect.flow_runs.worker - Completed submission of flow run '9d6ac137-9a6a-454d-b536-8c9d7600f163'
07:41:31.648 | INFO    | prefect.flow_runs.worker - Reported flow run '9d6ac137-9a6a-454d-b536-8c9d7600f163' as crashed: Flow run could not be submitted to infrastructure

11 Views

Open in Slack

Previous Next