https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> My prefect flow is reaching out of me...
# marvin-ai
t
@Marvin My prefect flow is reaching out of memory when installing packages:
Copy code
T
Downloading ruff
2025-09-30 11:23:16.000 CST
Downloading uv
2025-09-30 11:23:16.314 CST
Downloading mypy
2025-09-30 11:23:16.894 CST
Downloading polars
2025-09-30 11:23:17.521 CST
Downloading deltalake
2025-09-30 11:23:21.108 CST
Downloading msgraph-sdk
2025-09-30 11:23:22.424 CST
Installed 171 packages in 1.31s
2025-09-30 11:23:59.995 CST
Out-of-memory event detected in container
2025-09-30 11:24:00.081 CST
error: Failed to bytecode-compile Python file in: .venv/lib/python3.12/site-packages
2025-09-30 11:24:00.081 CST
Caused by: Bytecode compilation failed, expected "/opt/prefect/.venv/lib/python3.12/site-packages/msgraph/generated/identity_governance/lifecycle_workflows/deleted_items/workflows/item/task_reports/item/task_definition/task_definition_request_builder.py", received: ""
2025-09-30 11:24:00.130 CST
Container called exit(2).
2025-09-30 11:24:00.131 CST
E0930 17:24:00.130268 12 saferun.go:45] goroutine signal.exit exited with error: caught signal: terminated.
2025-09-30 11:24:00.131 CST
I0930 17:24:00.130347 40 ktdclient.go:966] Disabling Collectors
2025-09-30 11:24:00.131 CST
I0930 17:24:00.130383 40 krsieventreader.go:301] update probe enablement status (exec: false; mmap: false; exit: false; fork: false)
2025-09-30 11:24:00.131 CST
W0930 17:24:00.130451 40 saferun.go:47] goroutine ktd.Run exited because context is cancelled.
2025-09-30 11:24:00.131 CST
F0930 17:24:00.130654 1 watcher.go:273] Fatal error: ktdclient failed to start: caught signal: terminated
2025-09-30 11:24:00.553 CST
2025/09/30 17:24:00.553033 [ktd watcher] KTD agent process failed: exit status 1
2025-09-30 11:24:06.931 CST
Cloud RunRunJobxi4-dwimordene-b-feb4361c180c41e38bf84c78b8464cd6-kwhxv {@type: <http://type.googleapis.com/google.cloud.audit.AuditLog|type.googleapis.com/google.cloud.audit.AuditLog>,
Using cloud run gcp + prefect server. This is the flow image dockerfile:
Copy code
FROM prefecthq/prefect:3.4.6-python3.12
COPY ./src /opt/prefect/src
# COPY .venv /opt/prefect/.venv
COPY pyproject.toml /opt/prefect/pyproject.toml
COPY uv.lock /opt/prefect/uv.lock
COPY .python-version /opt/prefect/.python-version
WORKDIR /opt/prefect/
This is how I am deploying:
Copy code
okta_flow.deploy(
        name="okta-audit-logs-deployment",
        work_pool_name="client-wp",
        image="us-east1-docker.pkg.dev/company-self/ingestion-service-flow-image/dev:latest",
        build=False,
        push=False,
        job_variables={
            "service_account_name": "<mailto:prefect-worker-sa@client-prod.iam.gserviceaccount.com|prefect-worker-sa@client-prod.iam.gserviceaccount.com>",
            "region": "us-east1",
            "command": "uv run prefect flow-run execute",
        },
    )
m
✅ thought for 301.3 seconds
Thanks for the details — this looks like an out-of-memory during uv’s post-install bytecode compilation step inside your Cloud Run job. Two things are happening: 1) uv is syncing and installing your whole project at runtime (including dev tools like ruff/mypy), then compiling bytecode for all packages — that’s memory hungry. 2) Your Cloud Run job likely has the default/low memory (e.g., ~512 MiB–1 GiB), which isn’t enough for that compile step plus heavy libs like polars/deltalake/msgraph-sdk. Recommended fixes (pick 1 or combine): A) Pre-bake dependencies into the image and skip runtime sync - Build them during the Docker build so the runtime container doesn’t install/compile anything. - Then run your flow with Python directly (no uv sync at runtime). Example Dockerfile:
Copy code
FROM prefecthq/prefect:3.4.6-python3.12

WORKDIR /opt/prefect

# Copy lockfile/pyproject first to leverage Docker layer caching
COPY pyproject.toml /opt/prefect/pyproject.toml
COPY uv.lock /opt/prefect/uv.lock
# Optional: pin Python version used by uv
COPY .python-version /opt/prefect/.python-version

# Install project deps into the image at build time
# If you keep dev tools in a dev group, exclude them here (e.g., --no-dev or exclude dev group)
# If you don’t need bytecode compilation, disable it to cut memory use.
ENV UV_COMPILE_BYTECODE=0
RUN uv sync --frozen

# Now copy your code
COPY ./src /opt/prefect/src

# Default command can be set via your work pool/job or here
# CMD ["prefect", "flow-run", "execute"]
And in your deployment, change the command to avoid triggering a uv sync:
Copy code
job_variables={
  "service_account_name": "<mailto:prefect-worker-sa@client-prod.iam.gserviceaccount.com|prefect-worker-sa@client-prod.iam.gserviceaccount.com>",
  "region": "us-east1",
  "command": "prefect flow-run execute"
}
B) If you must install at runtime, give the job more memory and disable bytecode compilation - Increase Cloud Run memory and CPU in your work pool’s job variables. - Disable uv bytecode compilation to reduce peak memory during install. Example job_variables (Cloud Run worker):
Copy code
job_variables={
  "service_account_name": "<mailto:prefect-worker-sa@client-prod.iam.gserviceaccount.com|prefect-worker-sa@client-prod.iam.gserviceaccount.com>",
  "region": "us-east1",

  # Increase resources (tune as needed)
  "memory": "2Gi",
  "cpu": 2,

  # Reduce memory during install
  "env": {
    "UV_COMPILE_BYTECODE": "0"
  },

  # If you keep runtime installs, you can also try to avoid installing dev deps:
  # "command": "uv run --no-dev prefect flow-run execute"

  # Or, if you pre-bake, just run Prefect directly:
  "command": "prefect flow-run execute"
}
C) Keep dev tools out of the runtime environment - Don’t ship ruff/mypy in the image used to execute flows. Make sure they’re in a dev/optional group that you do not install in production (and avoid them during uv sync at runtime). - If you use uv groups, ensure you’re not installing the dev group on the job. For example, install only the default/runtime groups during Docker build, or use a “no dev” flag at install time. Why this helps - Pre-building dependencies avoids the heaviest memory spikes entirely. - Disabling bytecode compilation reduces peak memory after install. - Increasing memory/CPU lets your job successfully complete even if you must install at runtime. About the job variables - The Cloud Run worker supports setting command, env, cpu, and memory in job variables. You can set these per-deployment (as you’re doing) or in the work pool’s default job template via the UI. - For more on the GCP worker, see the guide: GCP worker guide A few clarifying questions to tailor this: - Are you using the “Cloud Run” or “Cloud Run v2” worker type for your work pool? - Do you want dependencies installed at runtime, or can we bake them into the image? - Are ruff/mypy intended for runtime? If not, can we move them to a dev group and exclude from production?
If you share your pyproject’s dependency groups, I can suggest precise uv commands to exclude dev/test deps and provide a tightened Dockerfile.
Open in Slack
PreviousNext
Powered by