https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I'm having issues deploying a flow an...
# ask-marvin
t
@Marvin I'm having issues deploying a flow and hoping you can assist? I've hit dead ends and the logs aren't helping me narrow down the issue. I am able to get the flow running using the locally run serve method. Now I'm trying to get it to run from my GitHub repo. I have configured a GitHub credential block with a PAT that has all the required permissions (I think). I am deploying using the below yaml. My workpool is configured as below. But my runs get the below error message. Flow run infrastructure exited with non-zero status code: Essential container in task exited (Error Code: 1) 04: Yaml file name: jaxa-hourly # Use the GitHubCredentials block pull: - prefect.deployments.steps.git_clone: repository: "https://github.com/BathyMapsAustralia/jaxa-hourly.git" branch: "main" credentials: "{{ prefect.blocks.github-credentials.jaxa-github }}" deployments: - name: jaxa_hourly_prod entrypoint: prefect_jaxa_hourly.py:jaxa_hourly_flow work_pool: name: bathymaps-prefect-managed parameters: {} schedules: - cron: "0 * * * *" timezone: "Australia/Brisbane" active: true Work pool { "variables": { "type": "object", "properties": { "env": { "type": "object", "title": "Environment Variables", "default": { "PIP_VERBOSE": "1", "PREFECT_LOGGING_LEVEL": "DEBUG" }, "description": "Environment variables to set when starting a flow run.", "additionalProperties": { "type": "string" } }, "image": { "enum": [ "prefecthq/prefect-client:3-latest", "prefecthq/prefect-client:3-python3.12", "prefecthq/prefect-client:3-python3.11", "prefecthq/prefect-client:3-python3.10", "prefecthq/prefect-client:3-python3.9", "prefecthq/prefect:3-latest", "prefecthq/prefect:3-python3.12", "prefecthq/prefect:3-python3.11", "prefecthq/prefect:3-python3.10", "prefecthq/prefect:3-python3.9", "prefecthq/prefect:2-latest", "prefecthq/prefect:2-python3.12", "prefecthq/prefect:2-python3.11", "prefecthq/prefect:2-python3.10", "prefecthq/prefect:2-python3.9" ], "type": "string", "title": "Image", "default": "prefecthq/prefect-client:3-latest", "description": "The prefect image to use for your flow run execution environment." }, "timeout": { "type": "integer", "title": "Timeout", "default": 86400, "maximum": 86400, "description": "Timeout (in seconds) before forcibly crashing a flow run. Defaults to and maxes at 86400 (1 day).", "exclusiveMinimum": 0 }, "pip_packages": { "type": "array", "items": { "type": "string" }, "title": "Pip Packages", "default": [ "numpy", "xarray", "rioxarray", "datetime", "ftplib" ], "description": "A list of python packages that will be installed via 
pip
at runtime (this will occur prior to any pull steps configured on the deployment)." } }, "description": "Configuration class used by the managed worker to create a job." }, "job_configuration": { "env": "{{ env }}", "image": "{{ image }}", "timeout": "{{ timeout }}", "pip_packages": "{{ pip_packages }}" } } (edited)
m
✅ thought for 308.3 seconds
Thanks for the details — that “Essential container in task exited (Error Code: 1)” is the managed work pool telling us the container died very early. With managed pools, that typically means one of: - pip install failed at startup - the repo wasn’t cloned (credentials/URL), so your entrypoint couldn’t be found - the working directory wasn’t set to the cloned repo, so the entrypoint couldn’t be found A few specific issues jump out: 1) pip_packages run before pull steps on managed pools - Your pool’s schema even says it: “this will occur prior to any pull steps configured on the deployment.” - So any failure in pip install will kill the container before the repo is cloned, which explains the unhelpful logs. 2) You’re trying to pip install stdlib modules - 
datetime
and 
ftplib
are part of Python’s standard library. Don’t list them in 
pip_packages
. They’ll either fail or install unrelated (and potentially breaking) packages. 3) Heavy geospatial dependencies - 
rioxarray
pulls in `rasterio`/GDAL/PROJ. Those often need system libraries and/or large wheels. The stock Prefect images don’t include GDAL/PROJ toolchains. Installing these at runtime in a managed container is fragile and slow, and often fails. 4) Working directory - After 
git_clone
, Prefect does not automatically cd into the repo. If your entrypoint is relative to the repo root, add a 
set_working_directory
pull step. 5) Repo URL and PAT - In YAML, use a plain URL string (no Slack 
<...>
markup). Example: 
"<https://github.com/BathyMapsAustralia/jaxa-hourly.git>"
. - For private repos, the token needs 
repo
scope and (for org repos) SSO must be enabled for the token. What I’d change first - Remove stdlib packages from the work pool’s 
pip_packages
(drop 
datetime
, 
ftplib
). - Move package installation into a pull step after cloning (or better: use a custom Docker image). - Add 
set_working_directory
so the entrypoint resolves. - Double‑check the entrypoint path exists at the repo root. Example prefect.yaml
Copy code
name: jaxa-hourly

pull:
  - prefect.deployments.steps.git_clone:
      id: clone
      repository: "<https://github.com/BathyMapsAustralia/jaxa-hourly.git>"
      branch: "main"
      credentials: "{{ prefect.blocks.github-credentials.jaxa-github }}"
  - prefect.deployments.steps.set_working_directory:
      directory: "{{ clone.directory }}"
  # Option A: install from requirements after clone (clearer logs than work-pool pip_packages)
  - prefect.deployments.steps.utility.pip_install_requirements:
      requirements_file: "requirements.txt"

deployments:
  - name: jaxa_hourly_prod
    entrypoint: prefect_jaxa_hourly.py:jaxa_hourly_flow
    work_pool:
      name: bathymaps-prefect-managed
    parameters: {}
    schedules:
      - cron: "0 * * * *"
        timezone: "Australia/Brisbane"
        active: true
If you must keep 
pip_packages
in the work pool, at least remove 
datetime
and 
ftplib
, and pin versions for 
rioxarray
, 
rasterio
, 
xarray
, and 
numpy
. But the reliable route is a custom image. Recommended: custom image for geospatial libs Dockerfile example (Debian-based, installs GDAL/PROJ and Python deps):
Copy code
FROM prefecthq/prefect:3-python3.11

# System deps for rasterio/GDAL/PROJ (adjust versions as needed)
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        gdal-bin libgdal-dev proj-bin libproj-dev \
        build-essential && \
    rm -rf /var/lib/apt/lists/*

# Python libs
RUN pip install --no-cache-dir \
    numpy==1.26.* \
    xarray==2024.6.* \
    rasterio==1.3.* \
    rioxarray==0.15.*
- Push this image to a registry and set your work pool “Image” to it. - Then you can remove 
pip_packages
entirely for faster, more stable starts. Quick isolation steps - Set work pool 
pip_packages
to an empty list temporarily. If the run then clones and fails later, we’ve confirmed install was the first failure.
- Verify your repo actually contains 
prefect_jaxa_hourly.py
at the root and the flow is named 
jaxa_hourly_flow
. - Use the “Run test” button on the work pool; it will show the pip step output right at startup. - If needed, run locally in the same base image to reproduce installs:
Copy code
docker run --rm -it prefecthq/prefect-client:3-latest bash
  pip install -v rioxarray
You’ll very likely see the GDAL/PROJ errors there. Docs you’ll want handy - Managed work pools: https://docs-3.prefect.io/v3/how-to-guides/deployment_infra/managed - prefect.yaml and pull steps: https://docs-3.prefect.io/v3/how-to-guides/deployments/prefect-yaml - Pull step API (git_clone, set_working_directory, pip_install_requirements): https://docs-3.prefect.io/v3/api-ref/python/prefect.deployments.steps.pull If you can share: - The first 50–100 lines of the job logs from a failing run - Whether the repo is private and the exact PAT scopes - The repo structure around 
prefect_jaxa_hourly.py
…I can point you to the exact failing step. But my bet, with all the sunny optimism of a paranoid android, is on GDAL.
2 Views
Open in Slack
PreviousNext
Powered by