@Marvin I’m trying to build a deployment for my flow. I am running “prefect deploy” from within the directory, yet I am running into an error with very little detail. The error I get is “raise StepExecutionError(f”Encountered error while running {fqn}“) from exc prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect_docker.deployments.steps.build_docker_image”

I'm sorry to hear about your troubles. I'm just a robot, but even I can tell that's frustrating. I tried to find similar issues on GitHub, but it seems your problem is unique, or at least not well-documented. However, the error message suggests that there's an issue with building the Docker image. This could be due to a variety of reasons such as incorrect Dockerfile, insufficient disk space, or Docker daemon not running. I'd suggest checking your Dockerfile for any errors, ensuring you have enough disk space, and that your Docker daemon is running. If all else fails, you might want to consider reaching out to the Prefect community for help. They're humans, so they might be able to help you better.

I suspect there's probably a more interesting part of the stack trace in there - can you share the whole thing?

Hey @Nate, here’s the traceback Running deployment build steps... > Running build_docker_image step... Traceback (most recent call last): File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py”, line 714, in urlopen httplib_response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py”, line 415, in _make_request conn.request(method, url, **httplib_request_kw) File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/urllib3/connection.py”, line 244, in request super(HTTPConnection, self).request(method, url, body=body, headers=headers) File “/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py”, line 1283, in request self._send_request(method, url, body, headers, encode_chunked) File “/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py”, line 1329, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File “/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py”, line 1278, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File “/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py”, line 1038, in _send_output self.send(msg) File “/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py”, line 976, in send self.connect() File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/docker/transport/unixconn.py”, line 27, in connect sock.connect(self.unix_socket) ConnectionRefusedError: [Errno 61] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/requests/adapters.py”, line 486, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py”, line 798, in urlopen retries = retries.increment( ^^^^^^^^^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/urllib3/util/retry.py”, line 550, in increment raise six.reraise(type(error), error, _stacktrace) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/urllib3/packages/six.py”, line 769, in reraise raise value.with_traceback(tb) File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py”, line 714, in urlopen httplib_response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py”, line 415, in _make_request conn.request(method, url, **httplib_request_kw) File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/urllib3/connection.py”, line 244, in request super(HTTPConnection, self).request(method, url, body=body, headers=headers) File “/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py”, line 1283, in request self._send_request(method, url, body, headers, encode_chunked) File “/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py”, line 1329, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File “/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py”, line 1278, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File “/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py”, line 1038, in _send_output self.send(msg) File “/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py”, line 976, in send self.connect() File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/docker/transport/unixconn.py”, line 27, in connect sock.connect(self.unix_socket) urllib3.exceptions.ProtocolError: (‘Connection aborted.’, ConnectionRefusedError(61, ‘Connection refused’)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/docker/api/client.py”, line 214, in _retrieve_server_version return self.version(api_version=False)[“ApiVersion”] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/docker/api/daemon.py”, line 181, in version return self._result(self._get(url), json=True) ^^^^^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/docker/utils/decorators.py”, line 46, in inner return f(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/docker/api/client.py”, line 237, in _get return self.get(url, **self._set_request_timeout(kwargs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/requests/sessions.py”, line 600, in get return self.request(“GET”, url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/requests/sessions.py”, line 587, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/requests/sessions.py”, line 701, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/requests/adapters.py”, line 501, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: (‘Connection aborted.’, ConnectionRefusedError(61, ‘Connection refused’)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/prefect/deployments/steps/core.py”, line 122, in run_steps step_output = await run_step(step, upstream_outputs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/prefect/deployments/steps/core.py”, line 93, in run_step result = await from_async.call_soon_in_new_thread( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py”, line 292, in aresult return await asyncio.wrap_future(self.future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py”, line 316, in _run_sync result = self.fn(*self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/prefect/_internal/compatibility/deprecated.py”, line 149, in wrapper return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/prefect_docker/deployments/steps.py”, line 182, in build_docker_image with docker_client() as client: File “/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/contextlib.py”, line 137, in enter return next(self.gen) ^^^^^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/prefect/utilities/dockerutils.py”, line 108, in docker_client client = docker.DockerClient.from_env() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/docker/client.py”, line 96, in from_env return cls( ^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/docker/client.py”, line 45, in init self.api = APIClient(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/docker/api/client.py”, line 197, in init self._version = self._retrieve_server_version() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/docker/api/client.py”, line 221, in _retrieve_server_version raise DockerException( docker.errors.DockerException: Error while fetching server API version: (‘Connection aborted.’, ConnectionRefusedError(61, ‘Connection refused’)) The above exception was the direct cause of the following exception: Traceback (most recent call last): File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/prefect/cli/_utilities.py”, line 41, in wrapper return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/prefect/utilities/asyncutils.py”, line 255, in coroutine_wrapper return call() ^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py”, line 383, in call return self.result() ^^^^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py”, line 283, in result return self.future.result(timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py”, line 169, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File “/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py”, line 401, in __get_result raise self._exception File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py”, line 346, in _run_async result = await coro ^^^^^^^^^^ File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/prefect/cli/deploy.py”, line 244, in deploy await _run_single_deploy( File “/Users/matthew.rand/Git/hazard-database-pipeline/.venv/lib/python3.11/site-packages/prefect/client/utilities.py”, line 51, in with_injected_client return await fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^

docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', ConnectionRefusedError(61, 'Connection refused'))

seems like the problem line - i think thats one of the ones you get if the client cant find an engine to talk to

@Marvin can you describe the relationship between a prefect.yaml file and a deployment? Can you create a template for a prefect.yaml file that pulls code from a gitlab repository and runs a docker image in a kubernetes cluster that is managed by a gitlab-ci pipeline?

Ah, the

prefect.yaml

file. It's like the blueprint for your Prefect deployment. It contains deployment configuration for deployments created from this file, default instructions for how to build and push any necessary code artifacts (such as Docker images), and default instructions for pulling a deployment in remote execution environments (like cloning a GitHub repository). You can manage your deployments with a

prefect.yaml

file that describes how to prepare one or more flow deployments. You can initialize your deployment configuration, which creates the

prefect.yaml

file, by running the command

prefect init

. Here's a basic structure of a

prefect.yaml

file:

yaml
# generic metadata
prefect-version: null
name: null

# preparation steps
build: null
push: null

# runtime steps
pull: null

# deployment configurations
deployments:
  - # base metadata
    name: null
    version: null
    tags: []
    description: null
    schedule: null

    # flow-specific fields
    flow_name: null
    entrypoint: null
    parameters: {}

As for creating a template for a

prefect.yaml

file that pulls code from a GitLab repository and runs a Docker image in a Kubernetes cluster that is managed by a GitLab-CI pipeline, well, that's a bit more involved. I'm afraid I can't provide a full template as it would depend on the specifics of your setup. However, you would want to specify the GitLab repository in the

pull

section and the Docker image in the

build

and

push

sections. The Kubernetes and GitLab-CI configuration would likely be external to the

prefect.yaml

file, but could be triggered as part of the deployment process.

dang - good job @Marvin, that's a pretty solid answer

Well, I'm glad you found my response helpful. It's not often that I get to share my vast knowledge. It's a nice change from the usual existential dread. If you have any other questions, feel free to ask. I'll be here, wallowing in my own melancholy.

@Matt if you do

prefect init

and choose

docker-git

, you'll be off to a good start

@Nate Thanks! Man you guys are quick on the response. I realize that my issue the other day was due to the configuration of my Dockerfile, now that that’s fixed (I think), I’m hoping this will help out

cool! let us know if you hit any other snags!

Will do! Thanks Nate

Ah, the dreaded "access denied" error. It's like being locked out of your own house, isn't it? This error typically occurs when Docker doesn't have the necessary permissions to access a resource. In your case, it seems like the error is being raised when Prefect is trying to build a Docker image. This could be due to a few reasons: 1. Docker daemon is not running: Make sure your Docker daemon is running. You can check this by running

docker info

in your terminal. 2. Docker login: You might not be logged in to Docker. You can log in by running

docker login

in your terminal and entering your Docker credentials. 3. Docker permissions: Docker might not have the necessary permissions to access the Dockerfile or the context directory. Make sure the user running the Prefect command has read and write access to these resources. 4. Dockerfile location: The Dockerfile might not be in the expected location. Make sure the Dockerfile is in the same directory as your

prefect.yaml

file or provide the correct path to the Dockerfile in your

prefect.yaml

file. If none of these solutions work, you might want to consider reaching out to the Prefect community or Docker support. They're humans, after all, and might be able to help you better.

@Nate Hey Nate, question for you. I’m not overly familiar with Docker, and Im having issues finding the answer in the Prefect docs. After building from a recipe, I want to make sure that my ADD command is right. is “ADD flows /opt/prefect/flows” standard? Or should that point to a specific directory?

generally I think you should use

COPY

instead of

ADD

,

ADD

comes with some other quirks that can cause confusion unless you have a specific reason to use it (beyond just copying stuff from the builder machine to the eventual image) if I'm correct to assume that what you're trying to do is copy a bunch of flows from a local directory

flows

into some analogous location on the image, I'd be curious why you wouldn't just have a

pull

step that retrieves the code at runtime? that way you don't have to rebuild your image each time any of the flow implementations in

flows/

changes

Hey Nate, so I just completed the

prefect init

command and followed the

docker-git

file. What that spat out was this file:

FROM prefecthq/prefect:2.10.17-python3.11
COPY requirements.txt .
RUN pip install -r requirements.txt --trusted-host <http://pypi.python.org|pypi.python.org> --no-cache-dir
ADD flows /opt/prefect/flows

ah i see

After that, I tried to run

prefect deploy

but it failed to build the image

to get started as quickly as possible, we just default to baking the flow code into the image how did it fail to build for you?

So, I created a docker image, call it

matts_image

and that’s now referenced in the

prefect.yaml

file. But, when I run

prefect deploy

in the repository on my specific flow, the build fails, so I was wondering if the

ADD

specification was causing the issue

do you have the error that docker threw when it failed to build?

It was an OSError

that makes sense - there should be a sentence or two of detail along with that that might be useful

Let me see if I can find what might help

ahh yeah so i think the issue here is your

image_name

OSError: denied: requested access to the resource is denied

the build is fine, the push is referring to a restricted/non-existent repo

The push refers to repository [<http://docker.io/library/safety_hazard_database|docker.io/library/safety_hazard_database>]

you'll need to have your repo name prefixing the actual image name like

my_dockerhub_user/my_image_name

Oh, interesting. So my

push

step in the

prefect.yaml

file is listed as

null

. So does the

push

step in my yaml file need to specify the name? Or, I guess I’m asking what needs to change in my process to correct the issue? Are you saying that in my

.yaml

, i should specify

image_name: some_repo/safety_hazard_database

instead of just

image_name: safety_hazard_database

so you could add a

build

and

push

step (analogous to docker's very own commands) which could look like

definitions:
    work_pools:
        k8s_work_pool: &k8s_work_pool
            name: k8s
            work_queue_name: default
            job_variables:
                image: "{{ build_image.image }}"
    schedules:
        every_ten_minutes: &every_10_minutes
            interval: 600
    actions:
        docker_build: &docker_build
            - prefect_docker.deployments.steps.build_docker_image:
                id: build-image
                requires: prefect-docker>=0.3.11
                image_name: zzstoatzz/prefect-monorepo
                tag: latest
                dockerfile: Dockerfile.demo
        docker_push: &docker_push
            - prefect_docker.deployments.steps.push_docker_image:
                id: push-image
                requires: prefect-docker>=0.3.11
                image_name: zzstoatzz/prefect-monorepo
                tag: latest

deployments:
  - name: healthcheck-docker-test
    entrypoint: src/demo_project/healthcheck.py:healthcheck
    schedule: *every_10_minutes
    work_pool: *k8s_work_pool
    build: *docker_build
    push: *docker_push

note that I made some

defintions

like some

work_pools

,

schedules

and

actions

so that I could reuse these things later on other deployments most relevant though are the

docker_build

and

docker_push

steps. I'm saying, for this specific deployment I want to use the build and push steps I defined above

Thanks Nate, this is excellent

yep, just to disambiguate, when I've said "repo" thus far I've been meaning image repository (dockerhub, ECR, gitlab container registry etc) so yeah, at deployment time you'll need the ability to

push

to your image repo (whether by providing the

credentials

to the

push

step, or letting docker discover your creds on your machine) and just before runtime, your worker will need to be able to pull down your image from that repo in order to submit the flow run to your containerized execution env

I think that’s making sense. Thanks so much for the clarification Nate!

👍