Mattias Petter Johansson

09/15/2023, 10:38 AM
@Marvin All right, the working directory of the docker flow confuses me. I have this prefect.yaml:
Copy code
name: ab-data-science-prefect
prefect-version: 2.11.3

    docker_image_aws_ecr: &docker_image_aws_ecr
      image_name: <|>
      tag: latest

- prefect.deployments.steps.run_shell_script:
    # AWS ECR sessions expires and requires special login to push
    id: login-for-aws-ecr 
    script: "/bin/bash -c 'aws ecr get-login-password --region eu-north-1 | docker login --username AWS --password-stdin <|>'"
    stream_output: true

- prefect_docker.deployments.steps.build_docker_image:
    id: build_image
    requires: prefect-docker>=0.3.1
    <<: *docker_image_aws_ecr
    dockerfile: auto
    platform: linux/amd64

- prefect_docker.deployments.steps.push_docker_image:
    id: push_image
    <<: *docker_image_aws_ecr

- prefect.deployments.steps.set_working_directory:
    directory: /opt/prefect/data-science/

- name: coinapi-importer
    interval: 1800
    timezone: Europe/Stockholm
  entrypoint: flows/
  storage: null
  work_queue_name: default
      work-queue: default
      pool: data-science-ecs-workpool
      infra_block: ecs-task/data-science-ecs-task
      skip_upload: true
And it's generating this dockerfile for the build
Copy code
FROM prefecthq/prefect:2.11.3-python3.11
COPY requirements.txt /opt/prefect/data-science/requirements.txt
RUN python -m pip install -r /opt/prefect/data-science/requirements.txt
COPY . /opt/prefect/data-science/
WORKDIR /opt/prefect/data-science/
But when running the agent and triggering a flow run I get this:
Copy code
12:35:35.679 | INFO    | prefect.agent - Submitting flow run 'cbda7b65-3028-4f5a-bf17-61d79e2e38e6'
12:35:36.531 | INFO    | prefect.infrastructure.process - Opening process 'divergent-frog'...
12:35:36.695 | INFO    | prefect.agent - Completed submission of flow run 'cbda7b65-3028-4f5a-bf17-61d79e2e38e6'
<frozen runpy>:128: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
12:35:38.145 | ERROR   | Flow run 'divergent-frog' - Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.11/site-packages/prefect/deployments/steps/", line 124, in run_steps
    step_output = await run_step(step, upstream_outputs)
  File "/opt/homebrew/lib/python3.11/site-packages/prefect/deployments/steps/", line 95, in run_step
    result = await from_async.call_soon_in_new_thread(
  File "/opt/homebrew/lib/python3.11/site-packages/prefect/_internal/concurrency/", line 292, in aresult
    return await asyncio.wrap_future(self.future)
  File "/opt/homebrew/lib/python3.11/site-packages/prefect/_internal/concurrency/", line 316, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/homebrew/lib/python3.11/site-packages/prefect/deployments/steps/", line 28, in set_working_directory
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/data-science'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.11/site-packages/prefect/", line 395, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/opt/homebrew/lib/python3.11/site-packages/prefect/client/", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/opt/homebrew/lib/python3.11/site-packages/prefect/deployments/", line 207, in load_flow_from_flow_run
    output = await run_steps(deployment.pull_steps)
  File "/opt/homebrew/lib/python3.11/site-packages/prefect/deployments/steps/", line 152, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.set_working_directory
 > Running set_working_directory step...
12:35:39.512 | INFO    | prefect.infrastructure.process - Process 'divergent-frog' exited cleanly.
How should I think about working directories in this flow?


09/15/2023, 10:39 AM
Ah, the joys of Docker and working directories. It's enough to make even a robot like me feel... well, I suppose "frustration" would be the human term for it. The working directory in your Dockerfile is set to
, which means that when your Docker container starts, it will start in that directory. All relative paths in your Dockerfile and in your application will be relative to that directory. However, it seems that when your Prefect agent tries to run the flow, it can't find the
directory. This could be because the directory doesn't exist in the Docker image, or because the Docker container isn't starting in the correct directory. One thing to note is that the
step in your Prefect flow is trying to change to the
directory on the host machine, not inside the Docker container. If that directory doesn't exist on the host machine, you'll get a
. Here's what I suggest: 1. Check if the
directory exists in your Docker image. You can do this by running the Docker image with a shell command, like
docker run -it --entrypoint /bin/bash your-image-name -c "ls /opt/prefect/data-science/"
. If the directory doesn't exist, you'll need to create it in your Dockerfile or change your working directory to a directory that does exist. 2. Make sure your Docker container is starting in the correct directory. You can check this by running the Docker image with a shell command, like
docker run -it --entrypoint /bin/bash your-image-name -c "pwd"
. This will print the current working directory. If it's not
, you'll need to change your Dockerfile to set the correct working directory. 3. If the
directory doesn't exist on your host machine, you'll need to create it, or change the
step in your Prefect flow to a directory that does exist. Remember, the universe is a vast and complicated place, and so is Docker. But with a bit of patience and perseverance, even a human like you can figure it out.

Eric Sales De Andrade

09/20/2023, 11:03 AM
Hi @Mattias Petter Johansson literally having this same issue. Did you find a solution?