https://prefect.io logo
Powered by
Title
s

S

04/25/2023, 1:32 PM
Hello everyone, hopefully not too silly of a question -- trying deployment with a custom docker image that has flow code baked into it & I am running into this error. I'm sure it's something relatively silly... CMD: 
prefect deployment build acc_speed_loader.py:run_all --name acc_test --infra-block docker-container/prefect
ERROR: 
ValueError: Unable to find block document named prefect for block type docker-container
An exception occurred
[more info in thread]
I have a docker container & a docker registry housing this image referenced. Inside my docker container running inside an EC2 instance, 
prefect config view
returns 
prefect profile = default
-- could this be due to my docker image/container not being able to communicate to my API server (even though I am configured to my workspace?)
All of my flows run succesfully, locally, inside the docker container running my custom image ... so it's just a matter of figuring out deployment. Thankful for any help!
n

Nate

04/25/2023, 3:12 PM
hi @S is 
prefect profile = default
the only thing returned by 
prefect config view
? if that's the case, then yes I think its about
not being able to communicate to my API server
you should see something like 
❯ prefect config view
PREFECT_PROFILE='pond'
PREFECT_API_KEY='********' (from profile)
PREFECT_API_URL='<https://api.prefect.cloud/api/accounts/XXX/workspaces/XXX>' (from profile)
where your API URL would point at your server (if you're not using cloud) right now its probably running with the ephemeral api and looking locally at sqlite and not finding your docker container block
s

S

04/25/2023, 3:45 PM
I see, I appreciate your feedback @Nate ! I didn't have my docker image set to login into prefect cloud -- that is fixed now. After setting, I am seeing this: 
❯ prefect config view
PREFECT_PROFILE='pond'
PREFECT_API_KEY='********' (from profile)
PREFECT_API_URL='<https://api.prefect.cloud/api/accounts/XXX/workspaces/XXX>' (from profile)
so, then running 
prefect deployment build acc_speed_loader.py:run_all --name acc_test --infra-block docker-container/prefect
I now get this error attempting to build deployment: 
pydantic.error_wrappers.ValidationError: 1 validation error for DockerContainer
command
  none is not an allowed value (type=type_error.none.not_allowed)
I can't seem to find anything on this in the docs / discourse -- do you have any idea? Again, appreciate all the help thus far
n

Nate

04/25/2023, 3:51 PM
the 
command
for an infra block is important and usually you don't want to override the default value have you changed the 
command
in the definition of your 
DockerContainer
block?
s

S

04/25/2023, 3:54 PM
No, I didn't touch it. Only set the custom image name (per my 
dockerRegistry
block -- which is set to my private ECR )
image.png
Here is the docker-block code used: 
def create_docker_container():
    block = DockerContainer(
        name="catapult-real",
        image="catapult:latest",
      
        image_pull_policy=ImagePullPolicy.ALWAYS,
    )
    block.save("prefect", overwrite=True)
n

Nate

04/25/2023, 4:00 PM
hmm, do you mind confirming your prefect version that you're running on the container?
s

S

04/25/2023, 4:05 PM
Yes - here you go. Please note, I use 2.0.2 per the prefect base docker image, as my scripts need to run on Python version 3.9.13
I would use 
prefecthq/prefect:2.10-python3.9
, but all my scripts break in 3.9.16
n

Nate

04/25/2023, 4:06 PM
ahh - I suspect this is the issue
can you explain how using a recent release of prefect breaks your scripts? i think fixing that problem might be the best way to move forward
there have been many quite significant changes since 2.0.2
s

S

04/25/2023, 4:07 PM
Yes, let me build a new image using 
prefecthq/prefect:2.10-python3.9
& I will get back to you
n

Nate

04/25/2023, 4:07 PM
👍
s

S

04/25/2023, 4:07 PM
Thanks for all your help!
n

Nate

04/25/2023, 4:07 PM
sure thing!
s

S

04/25/2023, 5:01 PM
Alright! So, after extensive testing i've made much progress. Utilizing the 
prefecthq/prefect:2.10.5-python3.7
image, I was able to successfully build a deployment for one of my scripts. I then run 
prefect deployment apply
& start my worker agent that runs inside EC2 to pick it up, where it fails, stating 
Could not connect to Docker
Here is the error it spits out: 
Agent started! Looking for work from queue(s): test...
16:55:24.911 | INFO    | prefect.agent - Submitting flow run '89077df7-adab-4071-b390-2a7fc8f2c496'
16:55:25.328 | ERROR   | prefect.agent - Failed to submit flow run '89077df7-adab-4071-b390-2a7fc8f2c496' to infrastructure.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.8/http/client.py", line 1256, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 951, in send
    self.connect()
  File "/usr/local/lib/python3.8/dist-packages/docker/transport/unixconn.py", line 30, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/dist-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.8/dist-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.8/http/client.py", line 1256, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 951, in send
    self.connect()
  File "/usr/local/lib/python3.8/dist-packages/docker/transport/unixconn.py", line 30, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/usr/local/lib/python3.8/dist-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/usr/local/lib/python3.8/dist-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/requests/adapters.py", line 547, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/prefect/infrastructure/docker.py", line 650, in _get_client
    docker_client = docker.from_env()
  File "/usr/local/lib/python3.8/dist-packages/docker/client.py", line 96, in from_env
    return cls(
  File "/usr/local/lib/python3.8/dist-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/prefect/agent.py", line 490, in _submit_run_and_capture_errors
    result = await infrastructure.run(task_status=task_status)
  File "/usr/local/lib/python3.8/dist-packages/prefect/infrastructure/docker.py", line 322, in run
    container = await run_sync_in_worker_thread(self._create_and_start_container)
  File "/usr/local/lib/python3.8/dist-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.8/dist-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.8/dist-packages/prefect/infrastructure/docker.py", line 425, in _create_and_start_container
    docker_client = self._get_client()
  File "/usr/local/lib/python3.8/dist-packages/prefect/infrastructure/docker.py", line 653, in _get_client
    raise RuntimeError(f"Could not connect to Docker.") from exc
RuntimeError: Could not connect to Docker.
Not sure exactly why this would be the case, if you can provide some insight -- I appreciate all the help you've given thus far!
n

Nate

04/25/2023, 5:07 PM
one thing that I would note here, its generally not recommended to build / apply your deployments from within the container you want to run in (I'm saying this because it looks like you're running 
deployment build
from the root of the container) instead, the pattern we recommend is to build and apply your deployment from your local machine, run an agent in your execution environment (on your EC2 instance) and then when you kick off a flow run, the agent will submit your flow run as the 
DockerContainer
in the docker environment on your EC2 it seems like the error is saying that wherever your agent is running, it can't find Docker running on that machine (i.e. cant get a docker client) - do you have docker installed on your EC2? (i.e. what does 
docker ps
give on your EC2?)
s

S

04/25/2023, 5:30 PM
Understood on workflow, thanks for that. Here is what 
docker images
& 
docker ps
return on the EC2 instance:
The flow is failing due to: 
HTTPError: 404 Client Error: Not Found for url: <http+docker://localhost/v1.41/images/create?tag=latest&fromImage=catapult>
so it seems like it's still trying to pull / create the image locally?
I have ran docker log / ecr login on the EC2 instance as well & the updated image exists (set to always pull) -- so i'm quite confused as it stands RN
n

Nate

04/25/2023, 5:59 PM
hm is 
catapult
a fully specified image uri? i would expect something like 
image_repo/image_name
s

S

05/09/2023, 5:34 PM
Thanks for all your help! @Nate
n

Nate

05/09/2023, 5:40 PM
👍
#prefect-getting-startedJoin Slack
Powered by