All right the working directory of the docker flow

Question

< Marvin> All right the working directory of the docker flow confuses me I have this prefect yaml ``` name ab data science prefect prefect version 2 11 3 definitions images docker image aws ecr amp docker image aws ecr image name tag latest build prefect deployments steps run shell script AWS ECR sessions expires and requires special login to push id login for aws ecr script bin bash c aws ecr get login password region eu north 1 | docker login username AWS password stdin stream output true prefect docker deployments steps build docker image id build image requires prefect docker gt =0 3 1 lt lt docker image aws ecr dockerfile auto platform linux amd64 prefect docker deployments steps push docker image id push image lt lt docker image aws ecr pull prefect deployments steps set working directory directory opt prefect data science deployments name coinapi importer schedule interval 1800 timezone Europe Stockholm entrypoint flows coinapi import aggregated py import coinapi rates aggregated storage null work queue name default execution ecs task work queue default pool data science ecs workpool infra block ecs task data science ecs task skip upload true``` And it s generating this dockerfile for the build ```FROM prefecthq prefect 2 11 3 python3 11 COPY requirements txt opt prefect data science requirements txt RUN python m pip install r opt prefect data science requirements txt COPY opt prefect data science WORKDIR opt prefect data science ``` But when running the agent and triggering a flow run I get this ```12 35 35 679 | INFO | prefect agent Submitting flow run cbda7b65 3028 4f5a bf17 61d79e2e38e6 12 35 36 531 | INFO | prefect infrastructure process Opening process divergent frog 12 35 36 695 | INFO | prefect agent Completed submission of flow run cbda7b65 3028 4f5a bf17 61d79e2e38e6 lt frozen runpy gt 128 RuntimeWarning prefect engine found in sys modules after import of package prefect but prior to execution of prefect engine this may result in unpredictable behaviour 12 35 38 145 | ERROR | Flow run divergent frog Flow could not be retrieved from deployment Traceback most recent call last File opt homebrew lib python3 11 site packages prefect deployments steps core py line 124 in run steps step output = await run step step upstream outputs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File opt homebrew lib python3 11 site packages prefect deployments steps core py line 95 in run step result = await from async call soon in new thread ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File opt homebrew lib python3 11 site packages prefect internal concurrency calls py line 292 in aresult return await asyncio wrap future self future ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File opt homebrew lib python3 11 site packages prefect internal concurrency calls py line 316 in run sync result = self fn self args self kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File opt homebrew lib python3 11 site packages prefect deployments steps pull py line 28 in set working directory os chdir directory FileNotFoundError Errno 2 No such file or directory opt prefect data science The above exception was the direct cause of the following exception Traceback most recent call last File opt homebrew lib python3 11 site packages prefect engine py line 395 in retrieve flow then begin flow run flow = await load flow from flow run flow run client=client ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File opt homebrew lib python3 11 site packages prefect client utilities py line 51 in with injected client return await fn args kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^ File opt homebrew lib python3 11 site packages prefect deployments deployments py line 207 in load flow from flow run output = await run steps deployment pull steps ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File opt homebrew lib python3 11 site packages prefect deployments steps core py line 152 in run steps raise StepExecutionError f Encountered error while running fqn from exc prefect deployments steps core StepExecutionError Encountered error while running prefect deployments steps set working directory gt Running set working directory step 12 35 39 512 | INFO | prefect infrastructure process Process divergent frog exited cleanly ``` How should I think about working directories in this flow

Marvin · Answer

Ah the joys of Docker and working directories It s enough to make even a robot like me feel well I suppose frustration would be the human term for it The working directory in your Dockerfile is set to ` opt prefect data science ` which means that when your Docker container starts it will start in that directory All relative paths in your Dockerfile and in your application will be relative to that directory However it seems that when your Prefect agent tries to run the flow it can t find the ` opt prefect data science ` directory This could be because the directory doesn t exist in the Docker image or because the Docker container isn t starting in the correct directory One thing to note is that the `set working directory` step in your Prefect flow is trying to change to the ` opt prefect data science ` directory on the host machine not inside the Docker container If that directory doesn t exist on the host machine you ll get a `FileNotFoundError` Here s what I suggest 1 Check if the ` opt prefect data science ` directory exists in your Docker image You can do this by running the Docker image with a shell command like `docker run it entrypoint bin bash your image name c ls opt prefect data science ` If the directory doesn t exist you ll need to create it in your Dockerfile or change your working directory to a directory that does exist 2 Make sure your Docker container is starting in the correct directory You can check this by running the Docker image with a shell command like `docker run it entrypoint bin bash your image name c pwd ` This will print the current working directory If it s not ` opt prefect data science ` you ll need to change your Dockerfile to set the correct working directory 3 If the ` opt prefect data science ` directory doesn t exist on your host machine you ll need to create it or change the `set working directory` step in your Prefect flow to a directory that does exist Remember the universe is a vast and complicated place and so is Docker But with a bit of patience and perseverance even a human like you can figure it out

Eric Sales De Andrade · Answer

Hi < Mattias Petter Johansson> literally having this same issue Did you find a solution