pradana adnyana
12/26/2025, 3:44 AMPATH
ENV PATH="/root/.local/bin/:$PATH"
COPY . /app
WORKDIR /app
RUN uv sync --locked
ENV PATH="/app/.venv/bin:$PATH"
why it raise that error @Marvin?Marvin
12/26/2025, 3:44 AMMarvin
12/26/2025, 3:44 AMpradana adnyana
12/26/2025, 3:45 AMPATH
ENV PATH="/root/.local/bin/:$PATH"
COPY . /app
WORKDIR /app
RUN uv sync --locked
ENV PATH="/app/.venv/bin:$PATH"
why it raise that error @Marvin?Marvin
12/26/2025, 3:45 AMMarvin
12/26/2025, 3:52 AMprefect deployment inspect "<flow-name>/<deployment-name>"
Look at:
- path — is it null/None?
- storage_document_id — is it null?
- pull_steps — are there any?
If path is null and there are no pull_steps, that’s the cause.
Ways to fix (pick one approach)
A) Code is baked into the image (no remote storage)
- Recreate the deployment so Prefect knows to load the flow from the local filesystem in the container.
- From your project root (the same directory you COPY into /app), run:
prefect deploy ./path/to/flow_file.py:flow_func_name -n my-deployment
This sets a local path for the deployment so crash hooks can reload the flow from inside the container.
- Or in Python:
flow.from_source(".", entrypoint="path/to/flow_file.py:flow_func_name").deploy(
name="my-deployment",
work_pool_name="your-pool"
)
B) Use pull steps (remote source)
- If your source is in Git, configure pull steps so the runner knows how to fetch code:
pull_steps:
- prefect.deployments.steps.set_working_directory:
directory: /app
- prefect.deployments.steps.git_clone:
repository: <https://github.com/<org>/<repo>>
branch: main
- Or use:
flow.from_source("<https://github.com/<org>/<repo>>", entrypoint="path/to/flow.py:flow").deploy(...)
C) Temporarily remove on_crashed hooks
- If you remove on_crashed hooks while you adjust the deployment, the runner won’t try to reload code after a crash and won’t hit this path.
A couple of extra checks
- Ensure your entrypoint string matches the file and function in the container (relative to WORKDIR /app), e.g. flows.py:my_flow.
- FYI: You’re on Python 3.13; Prefect’s latest stable support is typically behind the newest Python release. If you run into unrelated issues, try Python 3.11 or 3.12.
If you can share:
- How you created the deployment (CLI command or Python code),
- The output of prefect deployment inspect "<flow-name>/<deployment-name>" for path, storage_document_id, and pull_steps,
I can give you an exact minimal change to resolve it.pradana adnyana
12/26/2025, 3:57 AMMarvin
12/26/2025, 3:57 AMMarvin
12/26/2025, 3:59 AMprefect deploy workflows/main.py:main_flow -n "Google Maps Scraper Deployment"
Python API (explicitly set the source to the current project directory):
from workflows.main import main_flow
main_flow.from_source(".", entrypoint="workflows/main.py:main_flow").deploy(
name="Google Maps Scraper Deployment",
work_pool_name="docker-work-pool",
# Since your worker already sets the image via job variables, you can omit image here.
# If you do want Prefect to record it on the deployment:
# image="google-maps-scraper:latest",
# push=False, # keep image_pull_policy=Never behavior
)
Option B: Use a remote source so the runner knows how to fetch code
If your code lives in Git, define the source there so Prefect fetches it when needed:
from workflows.main import main_flow
main_flow.from_source(
"<https://github.com/<org>/<repo>>",
entrypoint="workflows/main.py:main_flow",
).deploy(
name="Google Maps Scraper Deployment",
work_pool_name="docker-work-pool",
)
Notes
- The initial crash still needs attention: your logs show the process exited with status code 1 before Prefect tried to reload the flow for crash hooks. Given you’re on python:3.13-slim, consider using Python 3.11 or 3.12 for now if you run into compatibility issues:
FROM python:3.12-slim
# ...rest of your Dockerfile...
- Your worker’s job_variables already specify image='google-maps-scraper:latest' and image_pull_policy='Never', so it will run whatever image is present on the worker node. Recreating the deployment with a path won’t change that behavior.
If you tell me how you originally created this deployment (CLI vs. Python) I can give you the minimal command or snippet to regenerate it so the path is set correctly.pradana adnyana
12/26/2025, 4:08 AMdocker run -it --rm google-maps-scraper ls -la
total 560
drwxr-xr-x 1 root root 4096 Dec 26 04:01 .
drwxr-xr-x 1 root root 4096 Dec 26 04:08 ..
-rw-r--r-- 1 root root 4589 Dec 26 03:43 .dockerignore
drwxr-xr-x 3 root root 4096 Dec 26 02:41 .github
-rw-r--r-- 1 root root 4571 Dec 19 03:31 .gitignore
-rw-r--r-- 1 root root 5 Dec 15 02:19 .python-version
drwxr-xr-x 5 root root 4096 Dec 26 04:02 .venv
-rw-r--r-- 1 root root 596 Dec 26 03:25 Dockerfile
-rw-r--r-- 1 root root 114 Dec 17 06:09 Makefile
-rw-r--r-- 1 root root 798 Dec 15 08:13 README.md
drwxr-xr-x 2 root root 4096 Dec 19 03:31 data
-rw-r--r-- 1 root root 91 Dec 15 03:02 main.py
drwxr-xr-x 2 root root 4096 Dec 19 03:31 notebooks
-rw-r--r-- 1 root root 426 Dec 26 02:50 pyproject.toml
drwxr-xr-x 5 root root 4096 Dec 15 08:13 src
drwxr-xr-x 2 root root 4096 Dec 17 08:09 storage
drwxr-xr-x 3 root root 4096 Dec 17 03:16 tests
-rw-r--r-- 1 root root 490172 Dec 26 02:50 uv.lock
drwxr-xr-x 3 root root 4096 Dec 18 02:18 workflowsMarvin
12/26/2025, 4:09 AMMarvin
12/26/2025, 4:11 AMworkflows/main.py:main_flow matches what’s in /app.
The remaining issue is your deployment still has:
- path: None
- storage_document_id: None
- pull_steps: None
With that combination, Prefect will try to download “from storage at None” after a crash and ends up looking in /app/None.
Let’s fix it by recreating the deployment so it records a local path.
Step-by-step
1) Delete the existing deployment (use the ID from your inspect)
prefect deployment delete --id 5a23e48a-a789-4ffc-81cf-427c610a48e8
2) From your project root (the same directory that gets copied to /app in the image), recreate the deployment so the path is set:
prefect deploy workflows/main.py:main_flow -n "Google Maps Scraper Deployment" --work-pool docker-work-pool
- This records a local path for the deployment based on your current working directory.
- Your worker already sets the image via job variables (google-maps-scraper:latest, image_pull_policy=Never), so you don’t need to pass an image to the deploy command.
3) Verify the fix
prefect deployment inspect "main_flow/Google Maps Scraper Deployment"
Confirm:
- path is not None (it should be a relative path like “.” or similar)
- storage_document_id is None (ok)
- pull_steps is None (ok)
Optional sanity check inside the container
docker run -it --rm google-maps-scraper /bin/sh -lc "cd /app && python -c 'from workflows.main import main_flow; print(main_flow.name)'"
This ensures your entrypoint imports correctly in the image.
If you prefer the Python API instead of the CLI, this achieves the same “local path” outcome:
from workflows.main import main_flow
main_flow.from_source(".", entrypoint="workflows/main.py:main_flow").deploy(
name="Google Maps Scraper Deployment",
work_pool_name="docker-work-pool",
)
If you still see the FileNotFoundError after redeploying, please paste the updated prefect deployment inspect (path/pull_steps fields) and, if possible, the logs from the initial crash (the exit code 1 cause). Also, if you continue to hit unrelated errors, try a Python 3.12 base image instead of 3.13:
FROM python:3.12-slimpradana adnyana
12/26/2025, 4:18 AMMarvin
12/26/2025, 4:18 AMMarvin
12/26/2025, 4:21 AMdocker run --rm -it \
-e PREFECT_API_URL=<your-api-url> \
-e PREFECT_API_KEY=<your-api-key> \
google-maps-scraper \
/bin/sh -lc 'cd /app && prefect deploy workflows/main.py:main_flow -n "Google Maps Scraper Deployment" --work-pool docker-work-pool'
Then inspect:
prefect deployment inspect "main_flow/Google Maps Scraper Deployment"
Confirm path now points to /app (or a relative path), not /home/pradana/...
Option 2: Use pull_steps (recommended if your code is in Git)
- Tell Prefect how to fetch the code into the container at runtime:
from workflows.main import main_flow
main_flow.from_source(
"<https://github.com/<org>/<repo>>",
entrypoint="workflows/main.py:main_flow",
).deploy(
name="Google Maps Scraper Deployment",
work_pool_name="docker-work-pool",
)
This avoids storing a host-specific path entirely.
Option 3: Use a module import entrypoint
- If your package is installed in the image (via your pyproject + uv), switch the entrypoint to a pure module import like:
- entrypoint: "workflows.main:main_flow"
- Prefect will import the flow rather than relying on the stored filesystem path.
Notes
- The env var PREFECT__STORAGE_BASE_PATH only helps when the deployment.path uses the $STORAGE_BASE_PATH placeholder, which is not the case when you deploy directly from a local directory. That’s why setting env vars won’t fix an absolute host path already stored in the deployment.
- If you stick with Option 1, remember to delete the old deployment first so you’re not accidentally using the one with the host path:
prefect deployment delete --id 5a23e48a-a789-4ffc-81cf-427c610a48e8
If you share how you prefer to manage code (baked in the image vs. Git), I’ll give you the exact command/snippet to regenerate the deployment so it uses /app instead of your host path.pradana adnyana
12/26/2025, 4:25 AMMarvin
12/26/2025, 4:25 AMMarvin
12/26/2025, 4:26 AM'pull_steps': [
{
'prefect.deployments.steps.set_working_directory': {
'directory': '/home/pradana/projects/outlet-scrape'
}
}
]
At runtime, the Docker container tries to cd into that path and fails because it only has your code under /app (your WORKDIR), not /home/pradana/....
Fix it by doing one of the following:
Option A: Keep pull_steps but point to the container path
- Recreate the deployment with the working directory set to `/app`:
- Delete the current deployment:
prefect deployment delete --id 9b0d0207-1672-480f-9d3a-763410f6dd19
- Recreate it so pull_steps uses `/app`:
- If you manage deployments via a config (prefect.yaml), set the pull step to /app and run prefect deploy.
- Otherwise, recreate via the same method you used before, but ensure the pull_steps set_working_directory uses /app.
Option B: Don’t use pull_steps and record a container-native path
- Create the deployment from inside your image so Prefect records /app (or a relative “.”) instead of your host path:
docker run --rm -it \
-e PREFECT_API_URL=<your-api-url> \
-e PREFECT_API_KEY=<your-api-key> \
google-maps-scraper \
/bin/sh -lc 'cd /app && prefect deploy workflows/main.py:main_flow -n "Google Maps Scraper Deployment" --work-pool docker-work-pool'
After either change, verify:
prefect deployment inspect "main_flow/Google Maps Scraper Deployment"
- If using pull_steps: the directory should be /app
- If not using pull_steps: path should not be None and should reflect a container path (e.g., “.” from /app)
Given your image layout and WORKDIR /app, the quickest fix is to change the pull_steps working directory to /app.pradana adnyana
12/26/2025, 4:27 AMfrom pathlib import Path
from loguru import logger
from prefect.docker import DockerImage
from workflows.main import main_flow
if __name__ == "__main__":
root_path = Path(__file__).parent.parent
# <http://logger.info|logger.info>(f"Deploying from root path: {root_path}")
# flow_to_deploy = flow.from_source(
# source=root_path,
# entrypoint="workflows/main.py:main_flow",
# )
docker_file_path = root_path / "Dockerfile"
<http://logger.info|logger.info>(f"Using Dockerfile at: {docker_file_path}")
if hasattr(main_flow, "deploy"):
main_flow.from_source(root_path, entrypoint="workflows.main:main_flow").deploy( # pyright: ignore[reportAttributeAccessIssue]
"Google Maps Scraper Deployment",
work_pool_name="docker-work-pool",
image=DockerImage(
name="google-maps-scraper", tag="latest", dockerfile=str(docker_file_path)
),
push=False,
job_variables={"image_pull_policy": "Never"}
)Marvin
12/26/2025, 4:27 AMMarvin
12/26/2025, 4:31 AMroot_path = Path(__file__).parent.parent
main_flow.from_source(root_path, entrypoint="workflows.main:main_flow").deploy(...)
- from_source(root_path, ...) with a local filesystem path creates a LocalStorage pull step that records the absolute path of root_path on your host (e.g., /home/pradana/projects/outlet-scrape).
- At runtime in Docker, Prefect tries to cd into that host path, which does not exist in the container (your code is at /app), so it fails.
What to change
Pick one of these fixes:
A) Deploy from inside your Docker image so the path becomes /app
- Update your script to use “.” as the source and avoid rebuilding the image during deploy:
# deploy.py
from workflows.main import main_flow
if __name__ == "__main__":
main_flow.from_source(".", entrypoint="workflows.main:main_flow").deploy(
name="Google Maps Scraper Deployment",
work_pool_name="docker-work-pool",
job_variables={"image_pull_policy": "Never", "image": "google-maps-scraper:latest"},
build=False,
push=False,
)
- Run it inside the image so Prefect records /app (WORKDIR) instead of your host path:
docker run --rm -it \
-e PREFECT_API_URL=<your-api-url> \
-e PREFECT_API_KEY=<your-api-key> \
google-maps-scraper \
/bin/sh -lc "cd /app && python deploy.py"
B) Use a Git source instead of a local filesystem path
- If your code is in Git, point from_source at the repo so Prefect uses pull steps to fetch code at runtime (no host path leakage):
from pathlib import Path
from loguru import logger
from prefect.docker import DockerImage
from workflows.main import main_flow
if __name__ == "__main__":
docker_file_path = Path(__file__).parent.parent / "Dockerfile"
<http://logger.info|logger.info>(f"Using Dockerfile at: {docker_file_path}")
main_flow.from_source(
"<https://github.com/<org>/<repo>>",
entrypoint="workflows.main:main_flow",
).deploy(
name="Google Maps Scraper Deployment",
work_pool_name="docker-work-pool",
image=DockerImage(
name="google-maps-scraper",
tag="latest",
dockerfile=str(docker_file_path),
),
push=False,
job_variables={"image_pull_policy": "Never"},
)
C) If you really want a local path, set it to the container path explicitly
- Do not pass a host path to from_source. Use "." and run the deploy inside the container (Option A). There isn’t a safe way to point from_source at a host path and have it remap to /app automatically later.
After changing the script, delete the old deployment and redeploy:
prefect deployment delete --id 9b0d0207-1672-480f-9d3a-763410f6dd19
# then run your updated deploy path (A or B above)
Why module entrypoint still needed
- You’re already using workflows.main:main_flow which is good; keep it. It avoids file-path sensitivity. The failure was due to the working directory pull step pointing to your host path, not the entrypoint itself.
If you choose Option A and still see issues, paste the new prefect deployment inspect ... so we can confirm:
- pull_steps is None, or
- pull_steps.set_working_directory.directory is “/app” (not your host path).pradana adnyana
12/26/2025, 4:44 AMMarvin
12/26/2025, 4:44 AMMarvin
12/26/2025, 4:44 AMfrom_source with a local filesystem path records your host path in the deployment’s pull steps, which won’t exist inside the container.
Two safe patterns going forward:
- If code is baked into the image: run the deploy inside the container and use "." as the source so the path resolves to /app
from workflows.main import main_flow
main_flow.from_source(".", entrypoint="workflows.main:main_flow").deploy(
name="Google Maps Scraper Deployment",
work_pool_name="docker-work-pool",
build=False,
push=False,
job_variables={"image_pull_policy": "Never", "image": "google-maps-scraper:latest"},
)
- If code is in Git: point from_source at the repo URL so Prefect pulls it at runtime
main_flow.from_source(
"<https://github.com/<org>/<repo>>",
entrypoint="workflows.main:main_flow",
).deploy(name="Google Maps Scraper Deployment", work_pool_name="docker-work-pool")
Docs for reference:
- Deployments (Prefect 3)
If you want, share how you prefer to manage code (image-baked vs Git) and I’ll give you an exact deploy snippet tailored to that setup.pradana adnyana
12/26/2025, 4:46 AMMarvin
12/26/2025, 4:46 AMMarvin
12/26/2025, 4:46 AMfrom_source.
Recommended pattern:
- Run the deploy inside the image so Prefect records /app (your WORKDIR), not a host path
- Call flow.deploy(...) directly (no storage/pull steps)
Example:
# deploy.py
from workflows.main import main_flow
if __name__ == "__main__":
main_flow.deploy(
name="Google Maps Scraper Deployment",
work_pool_name="docker-work-pool",
job_variables={"image_pull_policy": "Never", "image": "google-maps-scraper:latest"},
build=False,
push=False,
)
Run it inside your image:
docker run --rm -it \
-e PREFECT_API_URL=<your-api-url> \
-e PREFECT_API_KEY=<your-api-key> \
google-maps-scraper \
/bin/sh -lc "cd /app && python deploy.py"
Key points:
- Don’t use a host filesystem path during deploy; run the deploy in the same layout your container uses.
- If you prefer CLI, same idea: run prefect deploy workflows/main.py:main_flow inside the container.
If you ever do use from_source with baked code, only use "." and run it inside the container so it resolves to /app.pradana adnyana
12/26/2025, 5:22 AMMarvin
12/26/2025, 5:22 AMMarvin
12/26/2025, 5:23 AMdocker login <http://ghcr.io|ghcr.io> -u <github-username> -p <github-pat>
- Make sure the worker process can see those credentials:
- Docker stores creds in ~/.docker/config.json for the user who ran docker login.
- Run the worker as the same user OR set DOCKER_CONFIG so the worker can find the config:
- Example systemd unit:
[Service]
Environment=DOCKER_CONFIG=/home/prefect/.docker
ExecStart=/usr/local/bin/prefect worker start -p docker-work-pool
User=prefect
- Use the fully qualified image name in your deployment/work pool job variables:
job_variables={"image": "<http://ghcr.io/<owner>/<image>:<tag>|ghcr.io/<owner>/<image>:<tag>>", "image_pull_policy": "Always"}
- If you build/push images via Prefect deploy, log in on the machine running the deploy before calling deploy:
docker login <http://ghcr.io|ghcr.io> -u <github-username> -p <github-pat>
# then in Python
flow.deploy(image="<http://ghcr.io/<owner>/<image>:<tag>|ghcr.io/<owner>/<image>:<tag>>", push=True, ...)
If you use a Kubernetes work pool
- Create a docker-registry secret in the same namespace as your worker pods:
kubectl create secret docker-registry ghcr-creds \
--docker-server=<http://ghcr.io|ghcr.io> \
--docker-username=<github-username> \
--docker-password=<github-pat> \
--docker-email=<you@example.com> \
-n <your-namespace>
- Reference it so Prefect pods can pull the image:
- In the work pool’s job variables (UI or code), set image pull secrets to the secret name, and ensure your image is fully qualified:
job_variables={
"image": "<http://ghcr.io/<owner>/<image>:<tag>|ghcr.io/<owner>/<image>:<tag>>",
"image_pull_secrets": [{"name": "ghcr-creds"}]
}
- Or attach the secret to the ServiceAccount used by the worker so all pods inherit it:
kubectl patch serviceaccount <sa-name> \
-n <your-namespace> \
-p '{"imagePullSecrets": [{"name": "ghcr-creds"}]}'
Quick validation
- Docker host:
docker pull <http://ghcr.io/<owner>/<image>:<tag|ghcr.io/<owner>/<image>:<tag>>
- Kubernetes:
kubectl get secret ghcr-creds -n <ns>
kubectl run can-pull --image=<http://ghcr.io/<owner>/<image>:<tag|ghcr.io/<owner>/<image>:<tag>> -n <ns> --restart=Never
kubectl logs can-pull -n <ns>
Common gotchas
- Image must be named ghcr.io/<owner>/<image>:<tag> exactly
- Ensure the worker host (Docker) or the K8s nodes (Kubernetes) can reach ghcr.io
- For Docker workers: if you run the worker as a service user, that user must have a valid ~/.docker/config.json or DOCKER_CONFIG set
- For Kubernetes: the secret must exist in the same namespace where the pod runs, and be referenced in imagePullSecrets or ServiceAccount
If you share which work pool type you’re using (Docker vs Kubernetes) and how you’re deploying images (Prefect build/push vs prebuilt), I can provide tailored commands/snippets.pradana adnyana
12/26/2025, 5:24 AMMarvin
12/26/2025, 5:24 AMMarvin
12/26/2025, 5:25 AMdocker login <http://ghcr.io|ghcr.io> -u <github-username> -p <github-pat>
- Then run your deploy with an image like <http://ghcr.io/<owner>/<repo>:<tag|ghcr.io/<owner>/<repo>:<tag>> and push=True.
2) The machine(s) that pull and run the image (your workers/runtime)
- Docker work pool (Docker daemon on the worker host):
- Log in on the worker host:
docker login <http://ghcr.io|ghcr.io> -u <github-username> -p <github-pat>
- Ensure the worker process uses the same DOCKER_CONFIG (or same user) so it can pull.
- Set job_variables to use the GHCR image and a pull policy that actually pulls:
job_variables={
"image": "<http://ghcr.io/<owner>/<repo>:<tag>|ghcr.io/<owner>/<repo>:<tag>>",
"image_pull_policy": "IfNotPresent" # or "Always"
}
Note: If you keep "Never", the image must already be present on the worker.
- Kubernetes work pool:
- Create a docker-registry secret and reference it via imagePullSecrets or the ServiceAccount:
kubectl create secret docker-registry ghcr-creds \
--docker-server=<http://ghcr.io|ghcr.io> \
--docker-username=<github-username> \
--docker-password=<github-pat> \
--docker-email=<you@example.com> \
-n <ns>
Then set in job_variables:
job_variables={
"image": "<http://ghcr.io/<owner>/<repo>:<tag>|ghcr.io/<owner>/<repo>:<tag>>",
"image_pull_secrets": [{"name": "ghcr-creds"}]
}
The Prefect server/UI never builds or pulls images, so no login is needed there.
References:
- GitHub Container Registry docs
- Deployments (Prefect 3)pradana adnyana
12/26/2025, 5:26 AMMarvin
12/26/2025, 5:26 AMMarvin
12/26/2025, 5:26 AMdocker login <http://ghcr.io|ghcr.io> -u <github-username> -p <github-pat>
- Make sure the Prefect worker process uses the same Docker config (same user or set DOCKER_CONFIG):
export DOCKER_CONFIG=/home/prefect/.docker
prefect worker start -p docker-work-pool
- Or avoid login by pre-pulling the image and using `image_pull_policy: "Never"`:
docker pull <http://ghcr.io/<owner>/<image>:<tag|ghcr.io/<owner>/<image>:<tag>>
# or load from a tarball if you distribute images manually
docker load -i image.tar
Then set in your deployment/work pool:
job_variables={
"image": "<http://ghcr.io/<owner>/<image>:<tag>|ghcr.io/<owner>/<image>:<tag>>",
"image_pull_policy": "Never"
}
- Kubernetes work pool:
- Create an image pull secret and reference it (no docker login on nodes required):
kubectl create secret docker-registry ghcr-creds \
--docker-server=<http://ghcr.io|ghcr.io> \
--docker-username=<github-username> \
--docker-password=<github-pat> \
--docker-email=<you@example.com> \
-n <namespace>
Then in job variables:
job_variables={
"image": "<http://ghcr.io/<owner>/<image>:<tag>|ghcr.io/<owner>/<image>:<tag>>",
"image_pull_secrets": [{"name": "ghcr-creds"}]
}
- You do NOT need to log in on the Prefect server/UI host.
Notes
- GHCR creds: username = your GitHub username; password = a PAT with at least read:packages (and write:packages if you build/push).
- If your image is private and you don’t pre-pull it, the worker must be able to authenticate to ghcr.io.
Docs:
- GitHub Container Registry