S
04/25/2023, 1:32 PMprefect deployment build acc_speed_loader.py:run_all --name acc_test --infra-block docker-container/prefect
ERROR:
ValueError: Unable to find block document named prefect for block type docker-container
An exception occurred
[more info in thread]prefect config view
returns prefect profile = default
--
could this be due to my docker image/container not being able to communicate to my API server (even though I am configured to my workspace?)Nate
04/25/2023, 3:12 PMprefect profile = default
the only thing returned by prefect config view
? if that's the case, then yes I think its about
not being able to communicate to my API serveryou should see something like
❯ prefect config view
PREFECT_PROFILE='pond'
PREFECT_API_KEY='********' (from profile)
PREFECT_API_URL='<https://api.prefect.cloud/api/accounts/XXX/workspaces/XXX>' (from profile)
where your API URL would point at your server (if you're not using cloud)
right now its probably running with the ephemeral api and looking locally at sqlite and not finding your docker container blockS
04/25/2023, 3:45 PM❯ prefect config view
PREFECT_PROFILE='pond'
PREFECT_API_KEY='********' (from profile)
PREFECT_API_URL='<https://api.prefect.cloud/api/accounts/XXX/workspaces/XXX>' (from profile)
so, then running prefect deployment build acc_speed_loader.py:run_all --name acc_test --infra-block docker-container/prefect
I now get this error attempting to build deployment:
pydantic.error_wrappers.ValidationError: 1 validation error for DockerContainer
command
none is not an allowed value (type=type_error.none.not_allowed)
I can't seem to find anything on this in the docs / discourse -- do you have any idea?
Again, appreciate all the help thus farNate
04/25/2023, 3:51 PMcommand
for an infra block is important and usually you don't want to override the default value
have you changed the command
in the definition of your DockerContainer
block?S
04/25/2023, 3:54 PMdockerRegistry
block -- which is set to my private ECR )def create_docker_container():
block = DockerContainer(
name="catapult-real",
image="catapult:latest",
image_pull_policy=ImagePullPolicy.ALWAYS,
)
block.save("prefect", overwrite=True)
Nate
04/25/2023, 4:00 PMS
04/25/2023, 4:05 PMprefecthq/prefect:2.10-python3.9
, but all my scripts break in 3.9.16Nate
04/25/2023, 4:06 PMS
04/25/2023, 4:07 PMprefecthq/prefect:2.10-python3.9
& I will get back to youNate
04/25/2023, 4:07 PMS
04/25/2023, 4:07 PMNate
04/25/2023, 4:07 PMS
04/25/2023, 5:01 PMprefecthq/prefect:2.10.5-python3.7
image, I was able to successfully build a deployment for one of my scripts.
I then run prefect deployment apply
& start my worker agent that runs inside EC2 to pick it up, where it fails, stating Could not connect to Docker
Here is the error it spits out:
Agent started! Looking for work from queue(s): test...
16:55:24.911 | INFO | prefect.agent - Submitting flow run '89077df7-adab-4071-b390-2a7fc8f2c496'
16:55:25.328 | ERROR | prefect.agent - Failed to submit flow run '89077df7-adab-4071-b390-2a7fc8f2c496' to infrastructure.
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 398, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.8/http/client.py", line 1256, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output
self.send(msg)
File "/usr/lib/python3.8/http/client.py", line 951, in send
self.connect()
File "/usr/local/lib/python3.8/dist-packages/docker/transport/unixconn.py", line 30, in connect
sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.8/dist-packages/urllib3/util/retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.8/dist-packages/urllib3/packages/six.py", line 769, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 398, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.8/http/client.py", line 1256, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output
self.send(msg)
File "/usr/lib/python3.8/http/client.py", line 951, in send
self.connect()
File "/usr/local/lib/python3.8/dist-packages/docker/transport/unixconn.py", line 30, in connect
sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 214, in _retrieve_server_version
return self.version(api_version=False)["ApiVersion"]
File "/usr/local/lib/python3.8/dist-packages/docker/api/daemon.py", line 181, in version
return self._result(self._get(url), json=True)
File "/usr/local/lib/python3.8/dist-packages/docker/utils/decorators.py", line 46, in inner
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 237, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/requests/adapters.py", line 547, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/prefect/infrastructure/docker.py", line 650, in _get_client
docker_client = docker.from_env()
File "/usr/local/lib/python3.8/dist-packages/docker/client.py", line 96, in from_env
return cls(
File "/usr/local/lib/python3.8/dist-packages/docker/client.py", line 45, in __init__
self.api = APIClient(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 197, in __init__
self._version = self._retrieve_server_version()
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 221, in _retrieve_server_version
raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/prefect/agent.py", line 490, in _submit_run_and_capture_errors
result = await infrastructure.run(task_status=task_status)
File "/usr/local/lib/python3.8/dist-packages/prefect/infrastructure/docker.py", line 322, in run
container = await run_sync_in_worker_thread(self._create_and_start_container)
File "/usr/local/lib/python3.8/dist-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(
File "/usr/local/lib/python3.8/dist-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/usr/local/lib/python3.8/dist-packages/prefect/infrastructure/docker.py", line 425, in _create_and_start_container
docker_client = self._get_client()
File "/usr/local/lib/python3.8/dist-packages/prefect/infrastructure/docker.py", line 653, in _get_client
raise RuntimeError(f"Could not connect to Docker.") from exc
RuntimeError: Could not connect to Docker.
Not sure exactly why this would be the case, if you can provide some insight -- I appreciate all the help you've given thus far!Nate
04/25/2023, 5:07 PMdeployment build
from the root of the container)
instead, the pattern we recommend is to build and apply your deployment from your local machine, run an agent in your execution environment (on your EC2 instance) and then when you kick off a flow run, the agent will submit your flow run as the DockerContainer
in the docker environment on your EC2
it seems like the error is saying that wherever your agent is running, it can't find Docker running on that machine (i.e. cant get a docker client) - do you have docker installed on your EC2? (i.e. what does docker ps
give on your EC2?)S
04/25/2023, 5:30 PMdocker images
& docker ps
return on the EC2 instance:HTTPError: 404 Client Error: Not Found for url: <http+docker://localhost/v1.41/images/create?tag=latest&fromImage=catapult>
so it seems like it's still trying to pull / create the image locally?Nate
04/25/2023, 5:59 PMcatapult
a fully specified image uri? i would expect something like image_repo/image_name
S
05/09/2023, 5:34 PMNate
05/09/2023, 5:40 PM