Ritabrata Moitra
05/11/2023, 6:14 AMYou'll need Docker Engine installed and running on the same machine as your agent
My agent runs on EKS using the official helm chart, but fails with RuntimeError: Could not connect to Docker
Do I need to install DockerEngine separately on the agent?
2. Can someone point me towards resources on running flows embedded in ECR images through ECS? ( My agent would continue to run on EKS though )
Any help would be largely appreciated 🙏Nate
05/11/2023, 2:41 PMimage
uri on your infra block and then use that in your deployment
Worth noting that ECS workers are a thing now, workers being (newer) strongly-typed agents that we will recommend going forward, as they are designed to work nicely with work pools (which are also typed). And then you can use projects to define a deployment that overrides the image
in the ECS job configuration via ECSVariablesRitabrata Moitra
05/14/2023, 4:39 PMRuntimeError: Could not connect to Docker.
error.
The helm agents have a similar output -
04:34:46.199 | INFO | prefect.agent - Submitting flow run 'ec2bec70-7375-4664-a3e7-9e1a98f6ee94'
04:34:46.400 | ERROR | prefect.agent - Failed to submit flow run 'ec2bec70-7375-4664-a3e7-9e1a98f6ee94' to infrastructure.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 398, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.10/http/client.py", line 1283, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.10/http/client.py", line 1329, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.10/http/client.py", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.10/http/client.py", line 1038, in _send_output
self.send(msg)
File "/usr/local/lib/python3.10/http/client.py", line 976, in send
self.connect()
File "/usr/local/lib/python3.10/site-packages/docker/transport/unixconn.py", line 30, in connect
sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.10/site-packages/urllib3/packages/six.py", line 769, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 398, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.10/http/client.py", line 1283, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.10/http/client.py", line 1329, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.10/http/client.py", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.10/http/client.py", line 1038, in _send_output
self.send(msg)
File "/usr/local/lib/python3.10/http/client.py", line 976, in send
self.connect()
File "/usr/local/lib/python3.10/site-packages/docker/transport/unixconn.py", line 30, in connect
sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
return self.version(api_version=False)["ApiVersion"]
File "/usr/local/lib/python3.10/site-packages/docker/api/daemon.py", line 181, in version
return self._result(self._get(url), json=True)
File "/usr/local/lib/python3.10/site-packages/docker/utils/decorators.py", line 46, in inner
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/docker/api/client.py", line 237, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 547, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/docker.py", line 650, in _get_client
docker_client = docker.from_env()
File "/usr/local/lib/python3.10/site-packages/docker/client.py", line 96, in from_env
return cls(
File "/usr/local/lib/python3.10/site-packages/docker/client.py", line 45, in __init__
self.api = APIClient(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/docker/api/client.py", line 197, in __init__
self._version = self._retrieve_server_version()
File "/usr/local/lib/python3.10/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 490, in _submit_run_and_capture_errors
result = await infrastructure.run(task_status=task_status)
File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/docker.py", line 322, in run
container = await run_sync_in_worker_thread(self._create_and_start_container)
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(
File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/docker.py", line 425, in _create_and_start_container
docker_client = self._get_client()
File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/docker.py", line 653, in _get_client
raise RuntimeError("Could not connect to Docker.") from exc
RuntimeError: Could not connect to Docker.
04:34:46.411 | INFO | prefect.agent - Completed submission of flow run 'ec2bec70-7375-4664-a3e7-9e1a98f6ee94'
My docker-container infrastructure block exactly resembles the configuration provided here - https://docs.prefect.io/latest/guides/deployment/docker/
Could you folks give any hints on what I might be doing wrong? Would be grateful for any help 🙏Christopher Boyd
05/15/2023, 7:57 PMNate
05/15/2023, 8:06 PMChristopher Boyd
05/15/2023, 8:07 PMJafar A
05/15/2023, 8:58 PMRitabrata Moitra
05/16/2023, 11:13 AMdocker-container
infrastructure terminology.
What I basically want is to build a docker image from my code, and then execute it anywhere ( EKS / ECS … ). My understanding was that this is what the docker-container
infrastructure block facilitates, but looks like I was mistaken.
I am generally a bit averse about running flows directly on EKS ( or ECS for that matter ) because of the dependency issues. In our setup, we containerise all our workloads and then run the same on different infrastructure components ( EC2 / Fargate / Lambda etc. ), and I was trying to replicate the same.
How do you folks typically go about this?Christopher Boyd
05/16/2023, 12:18 PM