I m trying to build a deployment for my flow I am

Question

< Marvin> I m trying to build a deployment for my flow I am running prefect deploy from within the directory yet I am running into an error with very little detail The error I get is raise StepExecutionError f Encountered error while running fqn from exc prefect deployments steps core StepExecutionError Encountered error while running prefect docker deployments steps build docker image

Marvin · Answer

I m sorry to hear about your troubles I m just a robot but even I can tell that s frustrating I tried to find similar issues on GitHub but it seems your problem is unique or at least not well documented However the error message suggests that there s an issue with building the Docker image This could be due to a variety of reasons such as incorrect Dockerfile insufficient disk space or Docker daemon not running I d suggest checking your Dockerfile for any errors ensuring you have enough disk space and that your Docker daemon is running If all else fails you might want to consider reaching out to the Prefect community for help They re humans so they might be able to help you better

Nate · Answer

I suspect there s probably a more interesting part of the stack trace in there can you share the whole thing

Matt · Answer

Hey < Nate> here s the traceback Running deployment build steps gt Running build docker image step Traceback most recent call last File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages urllib3 connectionpool py line 714 in urlopen httplib response = self make request ^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages urllib3 connectionpool py line 415 in make request conn request method url httplib request kw File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages urllib3 connection py line 244 in request super HTTPConnection self request method url body=body headers=headers File opt homebrew Cellar python 3 11 3 11 3 Frameworks Python framework Versions 3 11 lib python3 11 http client py line 1283 in request self send request method url body headers encode chunked File opt homebrew Cellar python 3 11 3 11 3 Frameworks Python framework Versions 3 11 lib python3 11 http client py line 1329 in send request self endheaders body encode chunked=encode chunked File opt homebrew Cellar python 3 11 3 11 3 Frameworks Python framework Versions 3 11 lib python3 11 http client py line 1278 in endheaders self send output message body encode chunked=encode chunked File opt homebrew Cellar python 3 11 3 11 3 Frameworks Python framework Versions 3 11 lib python3 11 http client py line 1038 in send output self send msg File opt homebrew Cellar python 3 11 3 11 3 Frameworks Python framework Versions 3 11 lib python3 11 http client py line 976 in send self connect File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages docker transport unixconn py line 27 in connect sock connect self unix socket ConnectionRefusedError Errno 61 Connection refused During handling of the above exception another exception occurred Traceback most recent call last File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages requests adapters py line 486 in send resp = conn urlopen ^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages urllib3 connectionpool py line 798 in urlopen retries = retries increment ^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages urllib3 util retry py line 550 in increment raise six reraise type error error stacktrace ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages urllib3 packages six py line 769 in reraise raise value with traceback tb File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages urllib3 connectionpool py line 714 in urlopen httplib response = self make request ^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages urllib3 connectionpool py line 415 in make request conn request method url httplib request kw File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages urllib3 connection py line 244 in request super HTTPConnection self request method url body=body headers=headers File opt homebrew Cellar python 3 11 3 11 3 Frameworks Python framework Versions 3 11 lib python3 11 http client py line 1283 in request self send request method url body headers encode chunked File opt homebrew Cellar python 3 11 3 11 3 Frameworks Python framework Versions 3 11 lib python3 11 http client py line 1329 in send request self endheaders body encode chunked=encode chunked File opt homebrew Cellar python 3 11 3 11 3 Frameworks Python framework Versions 3 11 lib python3 11 http client py line 1278 in endheaders self send output message body encode chunked=encode chunked File opt homebrew Cellar python 3 11 3 11 3 Frameworks Python framework Versions 3 11 lib python3 11 http client py line 1038 in send output self send msg File opt homebrew Cellar python 3 11 3 11 3 Frameworks Python framework Versions 3 11 lib python3 11 http client py line 976 in send self connect File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages docker transport unixconn py line 27 in connect sock connect self unix socket urllib3 exceptions ProtocolError Connection aborted ConnectionRefusedError 61 Connection refused During handling of the above exception another exception occurred Traceback most recent call last File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages docker api client py line 214 in retrieve server version return self version api version=False ApiVersion ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages docker api daemon py line 181 in version return self result self get url json=True ^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages docker utils decorators py line 46 in inner return f self args kwargs ^^^^^^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages docker api client py line 237 in get return self get url self set request timeout kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages requests sessions py line 600 in get return self request GET url kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages requests sessions py line 587 in request resp = self send prep send kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages requests sessions py line 701 in send r = adapter send request kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages requests adapters py line 501 in send raise ConnectionError err request=request requests exceptions ConnectionError Connection aborted ConnectionRefusedError 61 Connection refused During handling of the above exception another exception occurred Traceback most recent call last File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect deployments steps core py line 122 in run steps step output = await run step step upstream outputs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect deployments steps core py line 93 in run step result = await from async call soon in new thread ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect internal concurrency calls py line 292 in aresult return await asyncio wrap future self future ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect internal concurrency calls py line 316 in run sync result = self fn self args self kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect internal compatibility deprecated py line 149 in wrapper return fn args kwargs ^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect docker deployments steps py line 182 in build docker image with docker client as client File opt homebrew Cellar python 3 11 3 11 3 Frameworks Python framework Versions 3 11 lib python3 11 contextlib py line 137 in enter return next self gen ^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect utilities dockerutils py line 108 in docker client client = docker DockerClient from env ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages docker client py line 96 in from env return cls ^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages docker client py line 45 in init self api = APIClient args kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages docker api client py line 197 in init self version = self retrieve server version ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages docker api client py line 221 in retrieve server version raise DockerException docker errors DockerException Error while fetching server API version Connection aborted ConnectionRefusedError 61 Connection refused The above exception was the direct cause of the following exception Traceback most recent call last File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect cli utilities py line 41 in wrapper return fn args kwargs ^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect utilities asyncutils py line 255 in coroutine wrapper return call ^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect internal concurrency calls py line 383 in call return self result ^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect internal concurrency calls py line 283 in result return self future result timeout=timeout ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect internal concurrency calls py line 169 in result return self get result ^^^^^^^^^^^^^^^^^^^ File opt homebrew Cellar python 3 11 3 11 3 Frameworks Python framework Versions 3 11 lib python3 11 concurrent futures base py line 401 in get result raise self exception File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect internal concurrency calls py line 346 in run async result = await coro ^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect cli deploy py line 244 in deploy await run single deploy File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect client utilities py line 51 in with injected client return await fn args kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^

Matt · Answer

File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect cli deploy py line 244 in deploy await run single deploy File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect client utilities py line 51 in with injected client return await fn args kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect cli deploy py line 436 in run single deploy await run steps build steps step outputs print function=app console print File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect deployments steps core py line 150 in run steps raise StepExecutionError f Encountered error while running fqn from exc prefect deployments steps core StepExecutionError Encountered error while running prefect docker deployments steps build docker image An exception occurred

Matt · Answer

< Nate> I didn t realize that the deployment pulled from Docker I started docker locally but ran into a server error I ll get with my team s Docker guru and see if I can find the answer I m not logged in and don t have any containers running maybe that s causing the issue

Nate · Answer

```docker errors DockerException Error while fetching server API version Connection aborted ConnectionRefusedError 61 Connection refused ``` seems like the problem line i think thats one of the ones you get if the client cant find an engine to talk to

Matt · Answer

< Marvin> can you describe the relationship between a prefect yaml file and a deployment Can you create a template for a prefect yaml file that pulls code from a gitlab repository and runs a docker image in a kubernetes cluster that is managed by a gitlab ci pipeline

Marvin · Answer

Ah the `prefect yaml` file It s like the blueprint for your Prefect deployment It contains deployment configuration for deployments created from this file default instructions for how to build and push any necessary code artifacts such as Docker images and default instructions for pulling a deployment in remote execution environments like cloning a GitHub repository You can manage your deployments with a `prefect yaml` file that describes how to prepare one or more flow deployments You can initialize your deployment configuration which creates the `prefect yaml` file by running the command `prefect init` Here s a basic structure of a `prefect yaml` file ```yaml generic metadata prefect version null name null preparation steps build null push null runtime steps pull null deployment configurations deployments base metadata name null version null tags description null schedule null flow specific fields flow name null entrypoint null parameters ``` As for creating a template for a `prefect yaml` file that pulls code from a GitLab repository and runs a Docker image in a Kubernetes cluster that is managed by a GitLab CI pipeline well that s a bit more involved I m afraid I can t provide a full template as it would depend on the specifics of your setup However you would want to specify the GitLab repository in the `pull` section and the Docker image in the `build` and `push` sections The Kubernetes and GitLab CI configuration would likely be external to the `prefect yaml` file but could be triggered as part of the deployment process

Nate · Answer

dang good job < Marvin> that s a pretty solid answer

Marvin · Answer

Well I m glad you found my response helpful It s not often that I get to share my vast knowledge It s a nice change from the usual existential dread If you have any other questions feel free to ask I ll be here wallowing in my own melancholy

Nate · Answer

< Matt> if you do `prefect init` and choose `docker git` you ll be off to a good start

Matt · Answer

< Nate> Thanks Man you guys are quick on the response I realize that my issue the other day was due to the configuration of my Dockerfile now that that s fixed I think I m hoping this will help out

Nate · Answer

cool let us know if you hit any other snags

Matt · Answer

Will do Thanks Nate

Matt · Answer

Hey < Marvin> any idea why my build might be failing I m running into an OSError File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect docker deployments steps py line 233 in build docker image raise OSError event error OSError denied requested access to the resource is denied Would this indicate an issue with my docker file

Marvin · Answer

Ah the dreaded access denied error It s like being locked out of your own house isn t it This error typically occurs when Docker doesn t have the necessary permissions to access a resource In your case it seems like the error is being raised when Prefect is trying to build a Docker image This could be due to a few reasons 1 Docker daemon is not running Make sure your Docker daemon is running You can check this by running `docker info` in your terminal 2 Docker login You might not be logged in to Docker You can log in by running `docker login` in your terminal and entering your Docker credentials 3 Docker permissions Docker might not have the necessary permissions to access the Dockerfile or the context directory Make sure the user running the Prefect command has read and write access to these resources 4 Dockerfile location The Dockerfile might not be in the expected location Make sure the Dockerfile is in the same directory as your `prefect yaml` file or provide the correct path to the Dockerfile in your `prefect yaml` file If none of these solutions work you might want to consider reaching out to the Prefect community or Docker support They re humans after all and might be able to help you better

Matt · Answer

< Nate> Hey Nate question for you I m not overly familiar with Docker and Im having issues finding the answer in the Prefect docs After building from a recipe I want to make sure that my ADD command is right is ADD flows opt prefect flows standard Or should that point to a specific directory

Nate · Answer

generally I think you should use `COPY` instead of `ADD` `ADD` comes with some other quirks that can cause confusion unless you have a specific reason to use it beyond just copying stuff from the builder machine to the eventual image if I m correct to assume that what you re trying to do is copy a bunch of flows from a local directory `flows` into some analogous location on the image I d be curious why you wouldn t just have a `pull` step that retrieves the code at runtime that way you don t have to rebuild your image each time any of the flow implementations in `flows ` changes

Matt · Answer

Hey Nate so I just completed the `prefect init` command and followed the `docker git` file What that spat out was this file ```FROM prefecthq prefect 2 10 17 python3 11 COPY requirements txt RUN pip install r requirements txt trusted host <http pypi python org|pypi python org> no cache dir ADD flows opt prefect flows```

Nate · Answer

ah i see

Matt · Answer

After that I tried to run `prefect deploy` but it failed to build the image

Nate · Answer

to get started as quickly as possible we just default to baking the flow code into the image how did it fail to build for you

Matt · Answer

So I created a docker image call it `matts image` and that s now referenced in the `prefect yaml` file But when I run `prefect deploy` in the repository on my specific flow the build fails so I was wondering if the `ADD` specification was causing the issue

Nate · Answer

do you have the error that docker threw when it failed to build

Matt · Answer

It was an OSError

Nate · Answer

that makes sense there should be a sentence or two of detail along with that that might be useful

Matt · Answer

Let me see if I can find what might help

Matt · Answer

So here s the top of the call ```Running deployment build steps gt Running build docker image step Step 1 5 FROM prefecthq prefect 2 10 17 python3 11 gt 2d96a38329f0 Step 2 5 COPY requirements txt gt Using cache gt 419adc86766c Step 3 5 RUN pip install r requirements txt trusted host no cache dir gt Using cache gt 951415cef7a3 Step 4 5 ADD Pipeline Pipeline gt Using cache gt b0c79eeed8d0 Step 5 5 LABEL io prefect version=2 10 17 gt Running in e9bfe3396f9e gt 35b47a4209bb Successfully built 35b47a4209bb The push refers to repository Preparing Preparing Preparing Preparing Preparing Preparing Preparing Preparing Preparing Preparing Preparing Preparing Preparing Preparing Preparing Preparing Preparing Preparing Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting```

Matt · Answer

```Traceback most recent call last File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect deployments steps core py line 122 in run steps step output = await run step step upstream outputs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect deployments steps core py line 93 in run step result = await from async call soon in new thread ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect internal concurrency calls py line 292 in aresult return await asyncio wrap future self future ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect internal concurrency calls py line 316 in run sync result = self fn self args self kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect internal compatibility deprecated py line 149 in wrapper return fn args kwargs ^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect docker deployments steps py line 233 in build docker image raise OSError event error OSError denied requested access to the resource is denied The above exception was the direct cause of the following exception Traceback most recent call last File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect cli utilities py line 41 in wrapper return fn args kwargs ^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect utilities asyncutils py line 255 in coroutine wrapper return call ^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect internal concurrency calls py line 383 in call return self result ^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect internal concurrency calls py line 283 in result return self future result timeout=timeout ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect internal concurrency calls py line 169 in result return self get result ^^^^^^^^^^^^^^^^^^^ File opt homebrew Cellar python 3 11 3 11 3 Frameworks Python framework Versions 3 11 lib python3 11 concurrent futures base py line 401 in get result raise self exception File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect internal concurrency calls py line 346 in run async result = await coro ^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect cli deploy py line 244 in deploy await run single deploy File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect client utilities py line 51 in with injected client return await fn args kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^ File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect cli deploy py line 436 in run single deploy await run steps build steps step outputs print function=app console print File Users matthew rand Git hazard database pipeline venv lib python3 11 site packages prefect deployments steps core py line 150 in run steps raise StepExecutionError f Encountered error while running fqn from exc prefect deployments steps core StepExecutionError Encountered error while running prefect docker deployments steps build docker image An exception occurred ```

Matt · Answer

^That s the full traceback

Nate · Answer

ahh yeah so i think the issue here is your `image name` ```OSError denied requested access to the resource is denied``` the build is fine the push is referring to a restricted non existent repo ```The push refers to repository <http docker io library safety hazard database|docker io library safety hazard database> ``` you ll need to have your repo name prefixing the actual image name like ```my dockerhub user my image name```

Matt · Answer

Oh interesting So my `push` step in the `prefect yaml` file is listed as `null` So does the `push` step in my yaml file need to specify the name Or I guess I m asking what needs to change in my process to correct the issue Are you saying that in my ` yaml` i should specify `image name some repo safety hazard database` instead of just `image name safety hazard database`

Nate · Answer

so you could add a `build` and `push` step analogous to docker s very own commands which could look like ```definitions work pools k8s work pool amp k8s work pool name k8s work queue name default job variables image build image image schedules every ten minutes amp every 10 minutes interval 600 actions docker build amp docker build prefect docker deployments steps build docker image id build image requires prefect docker gt =0 3 11 image name zzstoatzz prefect monorepo tag latest dockerfile Dockerfile demo docker push amp docker push prefect docker deployments steps push docker image id push image requires prefect docker gt =0 3 11 image name zzstoatzz prefect monorepo tag latest deployments name healthcheck docker test entrypoint src demo project healthcheck py healthcheck schedule every 10 minutes work pool k8s work pool build docker build push docker push``` note that I made some `defintions` like some `work pools` `schedules` and `actions` so that I could reuse these things later on other deployments most relevant though are the `docker build` and `docker push` steps I m saying for this specific deployment I want to use the build and push steps I defined above

Nate · Answer

in this case my repo is `zzstoatzz` and my image name is `prefect monorepo` so `image name zzstoatzz prefect monorepo`

Nate · Answer

also just to avoid any confusion I have a global `pull` step at the top of my `prefect yaml` that automatically applies to all deployments defined here like ```pull prefect deployments steps git clone amp clone repo repository <https github com zzstoatzz prefect monorepo> branch main access token null rest of file```

Matt · Answer

Thanks Nate this is excellent

Matt · Answer

I think I understand what s going on I need my Dockerfile to build an image that copies my complete Gitlab repository and then my prefect yaml file should connect to my gitlab container registry so that I can push pull from the container registry not my actual repo

Matt · Answer

I think that s what I m catching based on the examples you provided

Nate · Answer

yep just to disambiguate when I ve said repo thus far I ve been meaning image repository dockerhub ECR gitlab container registry etc so yeah at deployment time you ll need the ability to `push` to your image repo whether by providing the `credentials` to the `push` step or letting docker discover your creds on your machine and just before runtime your worker will need to be able to pull down your image from that repo in order to submit the flow run to your containerized execution env

Matt · Answer

I think that s making sense Thanks so much for the clarification Nate

Nate · Answer

+1