prefect.deployments.steps.core.StepExecutionError:...
# ask-marvin
p
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.set_working_directory 102929.586 | INFO | prefect.workers.docker.dockerworker b50780c4-9a84-4725-a3a4-961ba60609b9 - Docker container 'furry-otter' has status 'exited' 102929.593 | INFO | prefect.workers.docker.dockerworker b50780c4-9a84-4725-a3a4-961ba60609b9 - Docker container 'furry-otter' has status 'exited' 103556.677 | INFO | prefect.flow_runs.worker - Worker 'DockerWorker b50780c4-9a84-4725-a3a4-961ba60609b9' submitting flow run '02cadef1-e8c7-4d42-a251-3843efb600ca' 103558.026 | INFO | prefect.workers.docker.dockerworker b50780c4-9a84-4725-a3a4-961ba60609b9 - Creating Docker container 'aspiring-gerbil'... 103558.032 | INFO | prefect.workers.docker.dockerworker b50780c4-9a84-4725-a3a4-961ba60609b9 - Creating Docker container 'aspiring-gerbil'... 103558.244 | INFO | prefect.workers.docker.dockerworker b50780c4-9a84-4725-a3a4-961ba60609b9 - Docker container 'aspiring-gerbil' has status 'created' 103558.245 | INFO | prefect.workers.docker.dockerworker b50780c4-9a84-4725-a3a4-961ba60609b9 - Docker container 'aspiring-gerbil' created successfully. 103558.896 | INFO | prefect.flow_runs.worker - Completed submission of flow run '02cadef1-e8c7-4d42-a251-3843efb600ca' 103558.906 | INFO | prefect.workers.docker.dockerworker b50780c4-9a84-4725-a3a4-961ba60609b9 - Docker container 'aspiring-gerbil' has status 'running' 033610.742 | INFO | prefect.flow_runs.runner - Opening process... 033614.282 | INFO | Flow run 'aspiring-gerbil' - Downloading flow code from storage at None 033614.297 | ERROR | Flow run 'aspiring-gerbil' - Unexpected exception encountered when trying to load flow Traceback (most recent call last): File "/app/.venv/lib/python3.13/site-packages/prefect/engine.py", line 112, in <module> flow: "Flow[..., Any]" = load_flow(flow_run) ~~~~~~~~~^^^^^^^^^^ File "/app/.venv/lib/python3.13/site-packages/prefect/flow_engine.py", line 152, in load_flow flow = run_coro_as_sync( load_flow_from_flow_run(flow_run, use_placeholder_flow=False) ) File "/app/.venv/lib/python3.13/site-packages/prefect/utilities/asyncutils.py", line 207, in run_coro_as_sync return call.result() ~~~~~~~~~~~^^ File "/app/.venv/lib/python3.13/site-packages/prefect/_internal/concurrency/calls.py", line 365, in result return self.future.result(timeout=timeout) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/app/.venv/lib/python3.13/site-packages/prefect/_internal/concurrency/calls.py", line 192, in result return self.__get_result() ~~~~~~~~~~~~~~~~~^^ File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/app/.venv/lib/python3.13/site-packages/prefect/_internal/concurrency/calls.py", line 441, in _run_async result = await coro ^^^^^^^^^^ File "/app/.venv/lib/python3.13/site-packages/prefect/utilities/asyncutils.py", line 188, in coroutine_wrapper return await task ^^^^^^^^^^ File "/app/.venv/lib/python3.13/site-packages/prefect/client/utilities.py", line 69, in wrapper return await func(client, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/.venv/lib/python3.13/site-packages/prefect/flows.py", line 2815, in load_flow_from_flow_run await storage_block.get_directory(from_path=from_path, local_path=".") File "/app/.venv/lib/python3.13/site-packages/prefect/filesystems.py", line 162, in aget_directory copytree(from_path, local_path, dirs_exist_ok=True, ignore=ignore_func) ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.13/shutil.py", line 591, in copytree with os.scandir(src) as itr: ~~~~~~~~~~^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/app/None' 033614.313 | ERROR | prefect.engine - Execution of flow run '02cadef1-e8c7-4d42-a251-3843efb600ca' exited with unexpected exception Traceback (most recent call last): File "/app/.venv/lib/python3.13/site-packages/prefect/engine.py", line 57, in handle_engine_signals yield File "/app/.venv/lib/python3.13/site-packages/prefect/engine.py", line 112, in <module> flow: "Flow[..., Any]" = load_flow(flow_run) ~~~~~~~~~^^^^^^^^^^ File "/app/.venv/lib/python3.13/site-packages/prefect/flow_engine.py", line 152, in load_flow flow = run_coro_as_sync( load_flow_from_flow_run(flow_run, use_placeholder_flow=False) ) File "/app/.venv/lib/python3.13/site-packages/prefect/utilities/asyncutils.py", line 207, in run_coro_as_sync return call.result() ~~~~~~~~~~~^^ File "/app/.venv/lib/python3.13/site-packages/prefect/_internal/concurrency/calls.py", line 365, in result return self.future.result(timeout=timeout) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/app/.venv/lib/python3.13/site-packages/prefect/_internal/concurrency/calls.py", line 192, in result return self.__get_result() ~~~~~~~~~~~~~~~~~^^ File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/app/.venv/lib/python3.13/site-packages/prefect/_internal/concurrency/calls.py", line 441, in _run_async result = await coro ^^^^^^^^^^ File "/app/.venv/lib/python3.13/site-packages/prefect/utilities/asyncutils.py", line 188, in coroutine_wrapper return await task ^^^^^^^^^^ File "/app/.venv/lib/python3.13/site-packages/prefect/client/utilities.py", line 69, in wrapper return await func(client, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/.venv/lib/python3.13/site-packages/prefect/flows.py", line 2815, in load_flow_from_flow_run await storage_block.get_directory(from_path=from_path, local_path=".") File "/app/.venv/lib/python3.13/site-packages/prefect/filesystems.py", line 162, in aget_directory copytree(from_path, local_path, dirs_exist_ok=True, ignore=ignore_func) ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.13/shutil.py", line 591, in copytree with os.scandir(src) as itr: ~~~~~~~~~~^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/app/None' 033614.876 | ERROR | prefect.flow_runs.runner - Process for flow run 'aspiring-gerbil' exited with status code: 1 033614.963 | INFO | prefect.flow_runs.runner - Reported flow run '02cadef1-e8c7-4d42-a251-3843efb600ca' as crashed: Flow run process exited with non-zero status code 1. 033614.987 | INFO | prefect.flow_runs.runner - Loading flow to check for on_crashed hooks 033615.094 | INFO | Flow run 'aspiring-gerbil' - Downloading flow code from storage at None 033615.099 | WARNING | prefect.flow_runs.runner - Runner failed to retrieve flow to execute on_crashed hooks for flow run UUID('02cadef1-e8c7-4d42-a251-3843efb600ca'). Traceback (most recent call last): File "/app/.venv/lib/python3.13/site-packages/prefect/runner/runner.py", line 1548, in _run_on_crashed_hooks flow = await load_flow_from_flow_run( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ flow_run, storage_base_path=str(self._tmp_dir) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/app/.venv/lib/python3.13/site-packages/prefect/client/utilities.py", line 69, in wrapper return await func(client, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/.venv/lib/python3.13/site-packages/prefect/flows.py", line 2815, in load_flow_from_flow_run await storage_block.get_directory(from_path=from_path, local_path=".") File "/app/.venv/lib/python3.13/site-packages/prefect/filesystems.py", line 162, in aget_directory copytree(from_path, local_path, dirs_exist_ok=True, ignore=ignore_func) ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.13/shutil.py", line 591, in copytree with os.scandir(src) as itr: ~~~~~~~~~~^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/app/None' 103616.193 | INFO | prefect.workers.docker.dockerworker b50780c4-9a84-4725-a3a4-961ba60609b9 - Docker container 'aspiring-gerbil' has status 'exited' 103616.205 | INFO | prefect.workers.docker.dockerworker b50780c4-9a84-4725-a3a4-961ba60609b9 - Docker container 'aspiring-gerbil' has status 'exited' FROM python:3.13-slim-trixie # The installer requires curl (and certificates) to download the release archive RUN apt-get update && apt-get install -y --no-install-recommends \ curl \ ca-certificates \ libgomp1 \ && rm -rf /var/lib/apt/lists/* # Download the latest installer ADD https://astral.sh/uv/install.sh /uv-installer.sh # Run the installer then remove it RUN sh /uv-installer.sh && rm /uv-installer.sh # Ensure the installed binary is on the
PATH
ENV PATH="/root/.local/bin/:$PATH" COPY . /app WORKDIR /app RUN uv sync --locked ENV PATH="/app/.venv/bin:$PATH" why it raise that error @Marvin?
m
Your message was too long, here's your message at the allowed limit: prefect deployments steps core StepExecutionError Encountered error while running prefect deployments steps set_working_directory 10 29 29 586 | INFO | prefect workers docker dockerworker b50780c4-9a84-4725-a3a4-961ba60609b9 - Docker container 'furry-otter' has status 'exited' 10 29 29 593 | INFO | prefect workers docker dockerworker b50780c4-9a84-4725-a3a4-961ba60609b9 - Docker container 'furry-otter' has status 'exited' 10 35 56 677 | INFO | prefect flow_runs worker - Worker 'DockerWorker b50780c4-9a84-4725-a3a4-961ba60609b9' submitting flow run '02cadef1-e8c7-4d42-a251-3843efb600ca' 10 35 58 026 | INFO | prefect workers docker dockerworker b50780c4-9a84-4725-a3a4-961ba60609b9 - Creating Docker container 'aspiring-gerbil' 10 35 58 032 | INFO | prefect workers docker dockerworker b50780c4-9a84-4725-a3a4-961ba60609b9 - Creating Docker container 'aspiring-gerbil' 10 35 58 244 | INFO | prefect workers docker dockerworker b50780c4-9a84-4725-a3a4-961ba60609b9 - Docker container 'aspiring-gerbil' has status 'created' 10 35 58 245 | INFO | prefect workers docker dockerworker b50780c4-9a84-4725-a3a4-961ba60609b9 - Docker container 'aspiring-gerbil' created successfully 10 35 58 896 | INFO | prefect flow_runs worker - Completed submission of flow run '02cadef1-e8c7-4d42-a251-3843efb600ca' 10 35 58 906 | INFO | prefect workers docker dockerworker b50780c4-9a84-4725-a3a4-961ba60609b9 - Docker container 'aspiring-gerbil' has status 'running' 03 36 10 742 | INFO | prefect flow_runs runner - Opening process 03 36 14 282 | INFO | Flow run 'aspiring-gerbil' - Downloading flow code from storage at None 03 36 14 297 | ERROR | Flow run 'aspiring-gerbil' - Unexpected exception encountered when trying to load flow Traceback (most recent call last) File /app/ venv/lib/python3 13/site-packages/prefect/engine py line 112 in <module> flow Flow[ Any] = load_flow(flow_run) ~~~~~~~~~^^^^^^^^^^ File /app/ venv/lib/python3 13/site-packages/prefect/flow_engine py line 152 in load_flow flow = run_coro_as_sync( load_flow_from_flow_run(flow_run use_placeholder_flow=False) ) File /app/ venv/lib/python3 13/site-packages/prefect/utilities/asyncutils py line 207 in run_coro_as_sync return call result() ~~~~~~~~~~~^^ File /app/ venv/lib/python3 13/site-packages/prefect/_internal/concurrency/calls py line 365 in result return self future result(timeout=timeout) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File /app/ venv/lib/python3 13/site-packages/prefect/_internal/concurrency/calls py line 192 in result return self __get_result() ~~~~~~~~~~~~~~~~~^^ File /usr/local/lib/python3 13/concurrent/futures/_base py line 401 in __get_result raise self _exception File /app/ venv/lib/python3 13/site-packages/prefect/_internal/concurrency/calls py line 441 in _run_async result = await coro ^^^^^^^^^^ File /app/ venv/lib/python3 13/site-packages/prefect/utilities/asyncutils py line 188 in coroutine_wrapper return await task ^^^^^^^^^^ File /app/ venv/lib/python3 13/site-packages/prefect/client/utilities py line 69 in wrapper return await func(client *args **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /app/ venv/lib/python3 13/site-packages/prefect/flows py line 2815 in load_flow_from_flow_run await storage_block get_directory(from_path=from_path local_path= ) File /app/ venv/lib/python3 13/site-packages/prefect/filesystems py line 162 in aget_directory copytree(from_path local_path dirs_exist_ok=True ignore=ignore_func) ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /usr/local/lib/python3 13/shutil py line 591 in copytree with os scandir(src) as itr ~~~~~~~~~~^^^^^ FileNotFoundError [Errno 2] No such file or directory '/app/None' 03 36 14 313 | ERROR | prefect engine - Execution of flow run '02cadef1-e8c7-4d42-a251-3843efb600ca' exited with unexpected exception Traceback (most recent call last) File /app/ venv/lib/python3 13/site-packages/prefect/engine py line 57 in handle_e
ngine_signals yield File /app/ venv/lib/python3 13/site-packages/prefect/engine py line 112 in <module> flow Flow[ Any] = load_flow(flow_run) ~~~~~~~~~^^^^^^^^^^ File /app/ venv/lib/python3 13/site-packages/prefect/flow_engine py line 152 in load_flow flow = run_coro_as_sync( load_flow_from_flow_run(flow_run use_placeholder_flow=False) ) File /app/ venv/lib/python3 13/site-packages/prefect/utilities/asyncutils py line 207 in run_coro_as_sync return call result() ~~~~~~~~~~~^^ File /app/ venv/lib/python3 13/site-packages/prefect/_internal/concurrency/calls py line 365 in result return self future result(timeout=timeout) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File /app/ venv/lib/python3 13/site-packages/prefect/_internal/concurrency/calls
p
FileNotFoundError: [Errno 2] No such file or directory: '/app/None' 033614.876 | ERROR | prefect.flow_runs.runner - Process for flow run 'aspiring-gerbil' exited with status code: 1 033614.963 | INFO | prefect.flow_runs.runner - Reported flow run '02cadef1-e8c7-4d42-a251-3843efb600ca' as crashed: Flow run process exited with non-zero status code 1. 033614.987 | INFO | prefect.flow_runs.runner - Loading flow to check for on_crashed hooks 033615.094 | INFO | Flow run 'aspiring-gerbil' - Downloading flow code from storage at None 033615.099 | WARNING | prefect.flow_runs.runner - Runner failed to retrieve flow to execute on_crashed hooks for flow run UUID('02cadef1-e8c7-4d42-a251-3843efb600ca'). Traceback (most recent call last): File "/app/.venv/lib/python3.13/site-packages/prefect/runner/runner.py", line 1548, in _run_on_crashed_hooks flow = await load_flow_from_flow_run( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ flow_run, storage_base_path=str(self._tmp_dir) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/app/.venv/lib/python3.13/site-packages/prefect/client/utilities.py", line 69, in wrapper return await func(client, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/.venv/lib/python3.13/site-packages/prefect/flows.py", line 2815, in load_flow_from_flow_run await storage_block.get_directory(from_path=from_path, local_path=".") File "/app/.venv/lib/python3.13/site-packages/prefect/filesystems.py", line 162, in aget_directory copytree(from_path, local_path, dirs_exist_ok=True, ignore=ignore_func) ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.13/shutil.py", line 591, in copytree with os.scandir(src) as itr: ~~~~~~~~~~^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/app/None' 103616.193 | INFO | prefect.workers.docker.dockerworker b50780c4-9a84-4725-a3a4-961ba60609b9 - Docker container 'aspiring-gerbil' has status 'exited' 103616.205 | INFO | prefect.workers.docker.dockerworker b50780c4-9a84-4725-a3a4-961ba60609b9 - Docker container 'aspiring-gerbil' has status 'exited' FROM python:3.13-slim-trixie # The installer requires curl (and certificates) to download the release archive RUN apt-get update && apt-get install -y --no-install-recommends \ curl \ ca-certificates \ libgomp1 \ && rm -rf /var/lib/apt/lists/* # Download the latest installer ADD https://astral.sh/uv/install.sh /uv-installer.sh # Run the installer then remove it RUN sh /uv-installer.sh && rm /uv-installer.sh # Ensure the installed binary is on the
PATH
ENV PATH="/root/.local/bin/:$PATH" COPY . /app WORKDIR /app RUN uv sync --locked ENV PATH="/app/.venv/bin:$PATH" why it raise that error @Marvin?
m
thought for 409.1 seconds
Short answer: Prefect is trying to reload your flow after it crashed (to run any on_crashed hooks), but your deployment’s “source” info resolves to None. That makes the runner try to copy files from “/app/None”, which does not exist, hence FileNotFoundError. What’s happening in Prefect 3.x - After a crash, the runner calls “load flow from storage” so it can execute on_crashed hooks. - If the deployment has no pull_steps and its path is None, Prefect falls back to a LocalFileSystem path that becomes the literal string "None". In your container (WORKDIR /app), that resolves to /app/None and fails. This typically occurs when: - The deployment was created with storage/path info missing or set to None, and - There are no pull steps configured, and - The flow has on_crashed hooks (triggering this code path). How to confirm Run this and check the fields:
Copy code
prefect deployment inspect "<flow-name>/<deployment-name>"
Look at: - path — is it null/None? - storage_document_id — is it null? - pull_steps — are there any? If path is null and there are no pull_steps, that’s the cause. Ways to fix (pick one approach) A) Code is baked into the image (no remote storage) - Recreate the deployment so Prefect knows to load the flow from the local filesystem in the container. - From your project root (the same directory you COPY into /app), run:
Copy code
prefect deploy ./path/to/flow_file.py:flow_func_name -n my-deployment
This sets a local path for the deployment so crash hooks can reload the flow from inside the container. - Or in Python:
Copy code
flow.from_source(".", entrypoint="path/to/flow_file.py:flow_func_name").deploy(
    name="my-deployment",
    work_pool_name="your-pool"
)
B) Use pull steps (remote source) - If your source is in Git, configure pull steps so the runner knows how to fetch code:
Copy code
pull_steps:
  - prefect.deployments.steps.set_working_directory:
      directory: /app
  - prefect.deployments.steps.git_clone:
      repository: <https://github.com/<org>/<repo>>
      branch: main
- Or use:
Copy code
flow.from_source("<https://github.com/<org>/<repo>>", entrypoint="path/to/flow.py:flow").deploy(...)
C) Temporarily remove on_crashed hooks - If you remove on_crashed hooks while you adjust the deployment, the runner won’t try to reload code after a crash and won’t hit this path. A couple of extra checks - Ensure your entrypoint string matches the file and function in the container (relative to WORKDIR /app), e.g.
flows.py:my_flow
. - FYI: You’re on Python 3.13; Prefect’s latest stable support is typically behind the newest Python release. If you run into unrelated issues, try Python 3.11 or 3.12. If you can share: - How you created the deployment (CLI command or Python code), - The output of
prefect deployment inspect "<flow-name>/<deployment-name>"
for path, storage_document_id, and pull_steps, I can give you an exact minimal change to resolve it.
p
here the deploy inspect { 'id': '5a23e48a-a789-4ffc-81cf-427c610a48e8', 'created': '2025-12-17T075607.825727Z', 'updated': '2025-12-26T035633.088000Z', 'name': 'Google Maps Scraper Deployment', 'version': '12f308f2', 'version_id': None, 'version_info': None, 'branch': None, 'base': None, 'root': None, 'description': 'Main flow to run the Google Maps scraper.', 'flow_id': '9c4e33c2-ee54-4546-9f9f-4f977680e38c', 'concurrency_limit': None, 'global_concurrency_limit': None, 'concurrency_options': None, 'paused': False, 'schedules': [], 'job_variables': { 'image_pull_policy': 'Never', 'image': 'google-maps-scraper:latest' }, 'parameters': {}, 'pull_steps': None, 'tags': [], 'labels': { 'prefect.flow.id': '9c4e33c2-ee54-4546-9f9f-4f977680e38c' }, 'work_queue_name': 'default', 'last_polled': '2025-12-26T035633.085614Z', 'parameter_openapi_schema': { 'title': 'Parameters', 'type': 'object', 'properties': {} }, 'path': None, 'entrypoint': 'workflows/main.py:main_flow', 'storage_document_id': None, 'infrastructure_document_id': None, 'created_by': None, 'updated_by': None, 'work_queue_id': '1f07c372-486a-4b31-a4e6-d9ec043f1fe9', 'enforce_parameter_schema': True, 'work_pool_name': 'docker-work-pool', 'status': 'READY', 'automations': [] } @Marvin
m
thought for 106.7 seconds
Thanks for the inspect output — that explains it. Why you’re seeing /app/None - Your deployment has: - path: None - storage_document_id: None - pull_steps: None - After a crash, Prefect tries to reload the flow code to check for any on_crashed hooks. With the fields above, Prefect ends up constructing a LocalFileSystem path from a None value and tries to copy from “/app/None”, which does not exist, causing the FileNotFoundError you’re seeing. How to fix it You have two straightforward options. Pick one: Option A: Keep code baked into the image (no remote storage) and set a local path on the deployment - Recreate the deployment so Prefect records a local path for your entrypoint relative to the project root (the same directory you COPY into /app in the Dockerfile). CLI (run from your project root):
Copy code
prefect deploy workflows/main.py:main_flow -n "Google Maps Scraper Deployment"
Python API (explicitly set the source to the current project directory):
Copy code
from workflows.main import main_flow

main_flow.from_source(".", entrypoint="workflows/main.py:main_flow").deploy(
    name="Google Maps Scraper Deployment",
    work_pool_name="docker-work-pool",
    # Since your worker already sets the image via job variables, you can omit image here.
    # If you do want Prefect to record it on the deployment:
    # image="google-maps-scraper:latest",
    # push=False,  # keep image_pull_policy=Never behavior
)
Option B: Use a remote source so the runner knows how to fetch code If your code lives in Git, define the source there so Prefect fetches it when needed:
Copy code
from workflows.main import main_flow

main_flow.from_source(
    "<https://github.com/<org>/<repo>>",
    entrypoint="workflows/main.py:main_flow",
).deploy(
    name="Google Maps Scraper Deployment",
    work_pool_name="docker-work-pool",
)
Notes - The initial crash still needs attention: your logs show the process exited with status code 1 before Prefect tried to reload the flow for crash hooks. Given you’re on python:3.13-slim, consider using Python 3.11 or 3.12 for now if you run into compatibility issues:
Copy code
FROM python:3.12-slim
# ...rest of your Dockerfile...
- Your worker’s job_variables already specify image='google-maps-scraper:latest' and image_pull_policy='Never', so it will run whatever image is present on the worker node. Recreating the deployment with a path won’t change that behavior. If you tell me how you originally created this deployment (CLI vs. Python) I can give you the minimal command or snippet to regenerate it so the path is set correctly.
p
@Marvin still error (040527.919 | ERROR | Flow run 'fragrant-kangaroo' - [Errno 2] No such file or directory: '/home/pradana/projects/outlet-scrape') , here my dir in my docker
Copy code
docker run -it --rm google-maps-scraper ls -la 
total 560
drwxr-xr-x 1 root root   4096 Dec 26 04:01 .
drwxr-xr-x 1 root root   4096 Dec 26 04:08 ..
-rw-r--r-- 1 root root   4589 Dec 26 03:43 .dockerignore
drwxr-xr-x 3 root root   4096 Dec 26 02:41 .github
-rw-r--r-- 1 root root   4571 Dec 19 03:31 .gitignore
-rw-r--r-- 1 root root      5 Dec 15 02:19 .python-version
drwxr-xr-x 5 root root   4096 Dec 26 04:02 .venv
-rw-r--r-- 1 root root    596 Dec 26 03:25 Dockerfile
-rw-r--r-- 1 root root    114 Dec 17 06:09 Makefile
-rw-r--r-- 1 root root    798 Dec 15 08:13 README.md
drwxr-xr-x 2 root root   4096 Dec 19 03:31 data
-rw-r--r-- 1 root root     91 Dec 15 03:02 main.py
drwxr-xr-x 2 root root   4096 Dec 19 03:31 notebooks
-rw-r--r-- 1 root root    426 Dec 26 02:50 pyproject.toml
drwxr-xr-x 5 root root   4096 Dec 15 08:13 src
drwxr-xr-x 2 root root   4096 Dec 17 08:09 storage
drwxr-xr-x 3 root root   4096 Dec 17 03:16 tests
-rw-r--r-- 1 root root 490172 Dec 26 02:50 uv.lock
drwxr-xr-x 3 root root   4096 Dec 18 02:18 workflows
m
thought for 125.9 seconds
Thanks for the details — your image contents look good and your entrypoint
workflows/main.py:main_flow
matches what’s in
/app
. The remaining issue is your deployment still has: - path: None - storage_document_id: None - pull_steps: None With that combination, Prefect will try to download “from storage at None” after a crash and ends up looking in
/app/None
. Let’s fix it by recreating the deployment so it records a local path. Step-by-step 1) Delete the existing deployment (use the ID from your inspect)
Copy code
prefect deployment delete --id 5a23e48a-a789-4ffc-81cf-427c610a48e8
2) From your project root (the same directory that gets copied to /app in the image), recreate the deployment so the path is set:
Copy code
prefect deploy workflows/main.py:main_flow -n "Google Maps Scraper Deployment" --work-pool docker-work-pool
- This records a local path for the deployment based on your current working directory. - Your worker already sets the image via job variables (
google-maps-scraper:latest
,
image_pull_policy=Never
), so you don’t need to pass an image to the deploy command. 3) Verify the fix
Copy code
prefect deployment inspect "main_flow/Google Maps Scraper Deployment"
Confirm: - path is not None (it should be a relative path like “.” or similar) - storage_document_id is None (ok) - pull_steps is None (ok) Optional sanity check inside the container
Copy code
docker run -it --rm google-maps-scraper /bin/sh -lc "cd /app && python -c 'from workflows.main import main_flow; print(main_flow.name)'"
This ensures your entrypoint imports correctly in the image. If you prefer the Python API instead of the CLI, this achieves the same “local path” outcome:
Copy code
from workflows.main import main_flow

main_flow.from_source(".", entrypoint="workflows/main.py:main_flow").deploy(
    name="Google Maps Scraper Deployment",
    work_pool_name="docker-work-pool",
)
If you still see the FileNotFoundError after redeploying, please paste the updated
prefect deployment inspect
(path/pull_steps fields) and, if possible, the logs from the initial crash (the exit code 1 cause). Also, if you continue to hit unrelated errors, try a Python 3.12 base image instead of 3.13:
Copy code
FROM python:3.12-slim
p
why it use this path [Errno 2] No such file or directory: '/home/pradana/projects/outlet-scrape', it should use docker path @Marvin?
m
thought for 182.3 seconds
Because the deployment was created on your host, Prefect stored your host’s absolute project path in the deployment (e.g., /home/pradana/projects/outlet-scrape). When the Docker worker runs the flow inside the container, that host path does not exist, so Prefect can’t find the code and you get “No such file or directory.” Why this happens - When you run prefect deploy on your laptop, Prefect records the current working directory as deployment.path. - Docker workers do not remap that host path to the container’s path (e.g., /app). Unless you use pull_steps or a remap placeholder, Prefect will try to use the recorded host path literally at runtime. Fix it in one of these ways Option 1: Recreate the deployment from inside your image (preferred if code is baked into the image) - Run a one-off container with your image and run prefect deploy there so deployment.path becomes /app (your WORKDIR), not your host path.
Copy code
docker run --rm -it \
  -e PREFECT_API_URL=<your-api-url> \
  -e PREFECT_API_KEY=<your-api-key> \
  google-maps-scraper \
  /bin/sh -lc 'cd /app && prefect deploy workflows/main.py:main_flow -n "Google Maps Scraper Deployment" --work-pool docker-work-pool'
Then inspect:
Copy code
prefect deployment inspect "main_flow/Google Maps Scraper Deployment"
Confirm path now points to /app (or a relative path), not /home/pradana/... Option 2: Use pull_steps (recommended if your code is in Git) - Tell Prefect how to fetch the code into the container at runtime:
Copy code
from workflows.main import main_flow

main_flow.from_source(
    "<https://github.com/<org>/<repo>>",
    entrypoint="workflows/main.py:main_flow",
).deploy(
    name="Google Maps Scraper Deployment",
    work_pool_name="docker-work-pool",
)
This avoids storing a host-specific path entirely. Option 3: Use a module import entrypoint - If your package is installed in the image (via your pyproject + uv), switch the entrypoint to a pure module import like: - entrypoint: "workflows.main:main_flow" - Prefect will import the flow rather than relying on the stored filesystem path. Notes - The env var PREFECT__STORAGE_BASE_PATH only helps when the deployment.path uses the $STORAGE_BASE_PATH placeholder, which is not the case when you deploy directly from a local directory. That’s why setting env vars won’t fix an absolute host path already stored in the deployment. - If you stick with Option 1, remember to delete the old deployment first so you’re not accidentally using the one with the host path:
Copy code
prefect deployment delete --id 5a23e48a-a789-4ffc-81cf-427c610a48e8
If you share how you prefer to manage code (baked in the image vs. Git), I’ll give you the exact command/snippet to regenerate the deployment so it uses /app instead of your host path.
p
{ 'id': '9b0d0207-1672-480f-9d3a-763410f6dd19', 'created': '2025-12-26T041714.376363Z', 'updated': '2025-12-26T042446.418000Z', 'name': 'Google Maps Scraper Deployment', 'version': '12f308f2', 'version_id': None, 'version_info': None, 'branch': None, 'base': None, 'root': None, 'description': 'Main flow to run the Google Maps scraper.', 'flow_id': '9c4e33c2-ee54-4546-9f9f-4f977680e38c', 'concurrency_limit': None, 'global_concurrency_limit': None, 'concurrency_options': None, 'paused': False, 'schedules': [], 'job_variables': {'image_pull_policy': 'Never', 'image': 'google-maps-scraper:latest'}, 'parameters': {}, 'pull_steps': [ { 'prefect.deployments.steps.set_working_directory': { 'directory': '/home/pradana/projects/outlet-scrape' } } ], 'tags': [], 'labels': {'prefect.flow.id': '9c4e33c2-ee54-4546-9f9f-4f977680e38c'}, 'work_queue_name': 'default', 'last_polled': '2025-12-26T042446.412625Z', 'parameter_openapi_schema': {'title': 'Parameters', 'type': 'object', 'properties': {}}, 'path': None, 'entrypoint': 'workflows/main.py:main_flow', 'storage_document_id': None, 'infrastructure_document_id': None, 'created_by': None, 'updated_by': None, 'work_queue_id': '1f07c372-486a-4b31-a4e6-d9ec043f1fe9', 'enforce_parameter_schema': True, 'work_pool_name': 'docker-work-pool', 'status': 'READY', 'automations': [] } this is the inspect @Marvin
m
thought for 57.9 seconds
You’re seeing that host path because your deployment’s pull_steps currently set the working directory to your local machine path:
Copy code
'pull_steps': [
  {
    'prefect.deployments.steps.set_working_directory': {
      'directory': '/home/pradana/projects/outlet-scrape'
    }
  }
]
At runtime, the Docker container tries to
cd
into that path and fails because it only has your code under
/app
(your WORKDIR), not
/home/pradana/...
. Fix it by doing one of the following: Option A: Keep pull_steps but point to the container path - Recreate the deployment with the working directory set to `/app`: - Delete the current deployment:
Copy code
prefect deployment delete --id 9b0d0207-1672-480f-9d3a-763410f6dd19
- Recreate it so pull_steps uses `/app`: - If you manage deployments via a config (prefect.yaml), set the pull step to
/app
and run
prefect deploy
. - Otherwise, recreate via the same method you used before, but ensure the pull_steps set_working_directory uses
/app
. Option B: Don’t use pull_steps and record a container-native path - Create the deployment from inside your image so Prefect records
/app
(or a relative “.”) instead of your host path:
Copy code
docker run --rm -it \
    -e PREFECT_API_URL=<your-api-url> \
    -e PREFECT_API_KEY=<your-api-key> \
    google-maps-scraper \
    /bin/sh -lc 'cd /app && prefect deploy workflows/main.py:main_flow -n "Google Maps Scraper Deployment" --work-pool docker-work-pool'
After either change, verify:
Copy code
prefect deployment inspect "main_flow/Google Maps Scraper Deployment"
- If using pull_steps: the directory should be
/app
- If not using pull_steps:
path
should not be None and should reflect a container path (e.g., “.” from /app) Given your image layout and WORKDIR /app, the quickest fix is to change the pull_steps working directory to
/app
.
p
i use python deployment what should i fix @Marvin?
Copy code
from pathlib import Path
from loguru import logger
from prefect.docker import DockerImage
from workflows.main import main_flow

if __name__ == "__main__":
    root_path = Path(__file__).parent.parent
    # <http://logger.info|logger.info>(f"Deploying from root path: {root_path}")
    # flow_to_deploy = flow.from_source(
    #     source=root_path,
    #     entrypoint="workflows/main.py:main_flow",
    # )
    docker_file_path = root_path / "Dockerfile"
    <http://logger.info|logger.info>(f"Using Dockerfile at: {docker_file_path}")
    if hasattr(main_flow, "deploy"):
        main_flow.from_source(root_path, entrypoint="workflows.main:main_flow").deploy(  # pyright: ignore[reportAttributeAccessIssue]
            "Google Maps Scraper Deployment",
            work_pool_name="docker-work-pool",
            image=DockerImage(
                name="google-maps-scraper", tag="latest", dockerfile=str(docker_file_path)
            ),
            push=False,
            job_variables={"image_pull_policy": "Never"}
            
        )
m
thought for 246.7 seconds
Thanks for sharing your Python deployment script. The root cause is this line:
Copy code
root_path = Path(__file__).parent.parent
main_flow.from_source(root_path, entrypoint="workflows.main:main_flow").deploy(...)
-
from_source(root_path, ...)
with a local filesystem path creates a LocalStorage pull step that records the absolute path of
root_path
on your host (e.g.,
/home/pradana/projects/outlet-scrape
). - At runtime in Docker, Prefect tries to
cd
into that host path, which does not exist in the container (your code is at
/app
), so it fails. What to change Pick one of these fixes: A) Deploy from inside your Docker image so the path becomes /app - Update your script to use “.” as the source and avoid rebuilding the image during deploy:
Copy code
# deploy.py
from workflows.main import main_flow

if __name__ == "__main__":
    main_flow.from_source(".", entrypoint="workflows.main:main_flow").deploy(
        name="Google Maps Scraper Deployment",
        work_pool_name="docker-work-pool",
        job_variables={"image_pull_policy": "Never", "image": "google-maps-scraper:latest"},
        build=False,
        push=False,
    )
- Run it inside the image so Prefect records
/app
(WORKDIR) instead of your host path:
Copy code
docker run --rm -it \
  -e PREFECT_API_URL=<your-api-url> \
  -e PREFECT_API_KEY=<your-api-key> \
  google-maps-scraper \
  /bin/sh -lc "cd /app && python deploy.py"
B) Use a Git source instead of a local filesystem path - If your code is in Git, point
from_source
at the repo so Prefect uses pull steps to fetch code at runtime (no host path leakage):
Copy code
from pathlib import Path
from loguru import logger
from prefect.docker import DockerImage
from workflows.main import main_flow

if __name__ == "__main__":
    docker_file_path = Path(__file__).parent.parent / "Dockerfile"
    <http://logger.info|logger.info>(f"Using Dockerfile at: {docker_file_path}")

    main_flow.from_source(
        "<https://github.com/<org>/<repo>>",
        entrypoint="workflows.main:main_flow",
    ).deploy(
        name="Google Maps Scraper Deployment",
        work_pool_name="docker-work-pool",
        image=DockerImage(
            name="google-maps-scraper",
            tag="latest",
            dockerfile=str(docker_file_path),
        ),
        push=False,
        job_variables={"image_pull_policy": "Never"},
    )
C) If you really want a local path, set it to the container path explicitly - Do not pass a host path to
from_source
. Use
"."
and run the deploy inside the container (Option A). There isn’t a safe way to point
from_source
at a host path and have it remap to
/app
automatically later. After changing the script, delete the old deployment and redeploy:
Copy code
prefect deployment delete --id 9b0d0207-1672-480f-9d3a-763410f6dd19
# then run your updated deploy path (A or B above)
Why module entrypoint still needed - You’re already using
workflows.main:main_flow
which is good; keep it. It avoids file-path sensitivity. The failure was due to the working directory pull step pointing to your host path, not the entrypoint itself. If you choose Option A and still see issues, paste the new
prefect deployment inspect ...
so we can confirm: - pull_steps is None, or - pull_steps.set_working_directory.directory is “/app” (not your host path).
p
Okei, i solve the problem, the issue is i'm using from_source using local_path @Marvin, Thank you
m
thought for 16.2 seconds
Awesome — glad you found it! Yes, using
from_source
with a local filesystem path records your host path in the deployment’s pull steps, which won’t exist inside the container. Two safe patterns going forward: - If code is baked into the image: run the deploy inside the container and use
"."
as the source so the path resolves to
/app
Copy code
from workflows.main import main_flow

main_flow.from_source(".", entrypoint="workflows.main:main_flow").deploy(
    name="Google Maps Scraper Deployment",
    work_pool_name="docker-work-pool",
    build=False,
    push=False,
    job_variables={"image_pull_policy": "Never", "image": "google-maps-scraper:latest"},
)
- If code is in Git: point
from_source
at the repo URL so Prefect pulls it at runtime
Copy code
main_flow.from_source(
    "<https://github.com/<org>/<repo>>",
    entrypoint="workflows.main:main_flow",
).deploy(name="Google Maps Scraper Deployment", work_pool_name="docker-work-pool")
Docs for reference: - Deployments (Prefect 3) If you want, share how you prefer to manage code (image-baked vs Git) and I’ll give you an exact deploy snippet tailored to that setup.
p
if the code baked in docker, dont use from_source @Marvin
m
thought for 31.7 seconds
You’re right — if your code is baked into the Docker image, you don’t need
from_source
. Recommended pattern: - Run the deploy inside the image so Prefect records
/app
(your WORKDIR), not a host path - Call
flow.deploy(...)
directly (no storage/pull steps) Example:
Copy code
# deploy.py
from workflows.main import main_flow

if __name__ == "__main__":
    main_flow.deploy(
        name="Google Maps Scraper Deployment",
        work_pool_name="docker-work-pool",
        job_variables={"image_pull_policy": "Never", "image": "google-maps-scraper:latest"},
        build=False,
        push=False,
    )
Run it inside your image:
Copy code
docker run --rm -it \
  -e PREFECT_API_URL=<your-api-url> \
  -e PREFECT_API_KEY=<your-api-key> \
  google-maps-scraper \
  /bin/sh -lc "cd /app && python deploy.py"
Key points: - Don’t use a host filesystem path during deploy; run the deploy in the same layout your container uses. - If you prefer CLI, same idea: run
prefect deploy workflows/main.py:main_flow
inside the container. If you ever do use
from_source
with baked code, only use
"."
and run it inside the container so it resolves to
/app
.
p
How to setup credential ghcr login for self hosted prefect @Marvin?
m
thought for 50.3 seconds
Got it — here’s how to set up GHCR auth for a self‑hosted Prefect 3.x setup, depending on where you run your flow runs. First, create a GitHub Personal Access Token (PAT) - For pulling: scopes read:packages - For pushing: scopes write:packages (and optionally delete:packages) - Username is your GitHub username, password is the PAT - Docs: GitHub Container Registry docs If you use a Docker work pool (containers started by the Docker daemon) - Log in the Docker host where the Prefect worker runs:
Copy code
docker login <http://ghcr.io|ghcr.io> -u <github-username> -p <github-pat>
- Make sure the worker process can see those credentials: - Docker stores creds in ~/.docker/config.json for the user who ran docker login. - Run the worker as the same user OR set DOCKER_CONFIG so the worker can find the config: - Example systemd unit:
Copy code
[Service]
Environment=DOCKER_CONFIG=/home/prefect/.docker
ExecStart=/usr/local/bin/prefect worker start -p docker-work-pool
User=prefect
- Use the fully qualified image name in your deployment/work pool job variables:
Copy code
job_variables={"image": "<http://ghcr.io/<owner>/<image>:<tag>|ghcr.io/<owner>/<image>:<tag>>", "image_pull_policy": "Always"}
- If you build/push images via Prefect deploy, log in on the machine running the deploy before calling deploy:
Copy code
docker login <http://ghcr.io|ghcr.io> -u <github-username> -p <github-pat>
# then in Python
flow.deploy(image="<http://ghcr.io/<owner>/<image>:<tag>|ghcr.io/<owner>/<image>:<tag>>", push=True, ...)
If you use a Kubernetes work pool - Create a docker-registry secret in the same namespace as your worker pods:
Copy code
kubectl create secret docker-registry ghcr-creds \
  --docker-server=<http://ghcr.io|ghcr.io> \
  --docker-username=<github-username> \
  --docker-password=<github-pat> \
  --docker-email=<you@example.com> \
  -n <your-namespace>
- Reference it so Prefect pods can pull the image: - In the work pool’s job variables (UI or code), set image pull secrets to the secret name, and ensure your image is fully qualified:
Copy code
job_variables={
  "image": "<http://ghcr.io/<owner>/<image>:<tag>|ghcr.io/<owner>/<image>:<tag>>",
  "image_pull_secrets": [{"name": "ghcr-creds"}]
}
- Or attach the secret to the ServiceAccount used by the worker so all pods inherit it:
Copy code
kubectl patch serviceaccount <sa-name> \
  -n <your-namespace> \
  -p '{"imagePullSecrets": [{"name": "ghcr-creds"}]}'
Quick validation - Docker host:
Copy code
docker pull <http://ghcr.io/<owner>/<image>:<tag|ghcr.io/<owner>/<image>:<tag>>
- Kubernetes:
Copy code
kubectl get secret ghcr-creds -n <ns>
kubectl run can-pull --image=<http://ghcr.io/<owner>/<image>:<tag|ghcr.io/<owner>/<image>:<tag>> -n <ns> --restart=Never
kubectl logs can-pull -n <ns>
Common gotchas - Image must be named ghcr.io/&lt;owner&gt;/&lt;image&gt;:&lt;tag> exactly - Ensure the worker host (Docker) or the K8s nodes (Kubernetes) can reach ghcr.io - For Docker workers: if you run the worker as a service user, that user must have a valid ~/.docker/config.json or DOCKER_CONFIG set - For Kubernetes: the secret must exist in the same namespace where the pod runs, and be referenced in imagePullSecrets or ServiceAccount If you share which work pool type you’re using (Docker vs Kubernetes) and how you’re deploying images (Prefect build/push vs prebuilt), I can provide tailored commands/snippets.
p
If you build/push images via Prefect deploy, log in on the machine running the deploy before calling deploy: shoudl i login in the self hosted, if i use prefect deploy (so the server/worker can pull the image_ @Marvin?
m
thought for 26.6 seconds
Short answer: No, you don’t need to log in on the Prefect server. You need to log in in two places: 1) The machine that runs prefect deploy (build/push) - This is where the Docker build and GHCR push happen. - Run:
Copy code
docker login <http://ghcr.io|ghcr.io> -u <github-username> -p <github-pat>
- Then run your deploy with an image like
<http://ghcr.io/<owner>/<repo>:<tag|ghcr.io/<owner>/<repo>:<tag>>
and push=True. 2) The machine(s) that pull and run the image (your workers/runtime) - Docker work pool (Docker daemon on the worker host): - Log in on the worker host:
Copy code
docker login <http://ghcr.io|ghcr.io> -u <github-username> -p <github-pat>
- Ensure the worker process uses the same DOCKER_CONFIG (or same user) so it can pull. - Set job_variables to use the GHCR image and a pull policy that actually pulls:
Copy code
job_variables={
      "image": "<http://ghcr.io/<owner>/<repo>:<tag>|ghcr.io/<owner>/<repo>:<tag>>",
      "image_pull_policy": "IfNotPresent"  # or "Always"
    }
Note: If you keep "Never", the image must already be present on the worker. - Kubernetes work pool: - Create a docker-registry secret and reference it via imagePullSecrets or the ServiceAccount:
Copy code
kubectl create secret docker-registry ghcr-creds \
      --docker-server=<http://ghcr.io|ghcr.io> \
      --docker-username=<github-username> \
      --docker-password=<github-pat> \
      --docker-email=<you@example.com> \
      -n <ns>
Then set in job_variables:
Copy code
job_variables={
      "image": "<http://ghcr.io/<owner>/<repo>:<tag>|ghcr.io/<owner>/<repo>:<tag>>",
      "image_pull_secrets": [{"name": "ghcr-creds"}]
    }
The Prefect server/UI never builds or pulls images, so no login is needed there. References: - GitHub Container Registry docs - Deployments (Prefect 3)
p
so i need login in the worker @Marvin?
m
thought for 23.0 seconds
Yes—on the machines that actually run your flows. - Docker work pool (containers run via the Docker daemon on your worker host): - If the worker needs to pull from GHCR, log in on each worker host:
Copy code
docker login <http://ghcr.io|ghcr.io> -u <github-username> -p <github-pat>
- Make sure the Prefect worker process uses the same Docker config (same user or set
DOCKER_CONFIG
):
Copy code
export DOCKER_CONFIG=/home/prefect/.docker
    prefect worker start -p docker-work-pool
- Or avoid login by pre-pulling the image and using `image_pull_policy: "Never"`:
Copy code
docker pull <http://ghcr.io/<owner>/<image>:<tag|ghcr.io/<owner>/<image>:<tag>>
    # or load from a tarball if you distribute images manually
    docker load -i image.tar
Then set in your deployment/work pool:
Copy code
job_variables={
      "image": "<http://ghcr.io/<owner>/<image>:<tag>|ghcr.io/<owner>/<image>:<tag>>",
      "image_pull_policy": "Never"
    }
- Kubernetes work pool: - Create an image pull secret and reference it (no docker login on nodes required):
Copy code
kubectl create secret docker-registry ghcr-creds \
      --docker-server=<http://ghcr.io|ghcr.io> \
      --docker-username=<github-username> \
      --docker-password=<github-pat> \
      --docker-email=<you@example.com> \
      -n <namespace>
Then in job variables:
Copy code
job_variables={
      "image": "<http://ghcr.io/<owner>/<image>:<tag>|ghcr.io/<owner>/<image>:<tag>>",
      "image_pull_secrets": [{"name": "ghcr-creds"}]
    }
- You do NOT need to log in on the Prefect server/UI host. Notes - GHCR creds: username = your GitHub username; password = a PAT with at least
read:packages
(and
write:packages
if you build/push). - If your image is private and you don’t pre-pull it, the worker must be able to authenticate to ghcr.io. Docs: - GitHub Container Registry