Thanks for the details — 40s is definitely on the high side, especially if the image is already on the host. From the timestamps you shared: - 100453 -> 100514 (≈21s): submission to “Opening process…” — typically container create/start + any image pull/auth checks - 100514 -> 100531 (≈17s): engine boot/imports/network handshake before “set_working_directory” - 100531 -> 100534 (3s): engine steps to begin the run Below are the fastest wins you can try, followed by a quick way to pinpoint where the time is going. Quick wins to reduce startup time 1) Skip image pull checks - If your image is preloaded, make sure the Docker work pool/job uses

image_pull_policy: IfNotPresent

or even

Never

for testing. If it’s

Always

, the worker still attempts a pull (auth + registry handshake) which can easily cost ~10–20s even with a cache hit. - Set this via deployment job variables or the pool’s base job template. Example (Python API during deploy):

flow.from_source(".").deploy(
  name="my-deploy",
  work_pool_name="docker-pool",
  image="your-repo/your-image:tag",
  job_variables={
    "image_pull_policy": "IfNotPresent"
  },
)

2) Trim Python/import startup inside the container - Use a smaller, faster base: e.g., python:3.11-slim, no conda activation on start. - Avoid heavy top-level imports in your flow module; move them inside tasks or

if __name__ == "__main__":

blocks. - Consider setting: -

ENV PYTHONDONTWRITEBYTECODE=1

(avoid writing .pyc at runtime) -

ENV PYTHONUNBUFFERED=1

- Ensure your entrypoint/cmd isn’t wrapped by a slow shell or conda init script. - Keep your image lean (multi-stage builds, clean pip/apt caches). 3) Minimize network/DNS overhead - The engine connects to the Prefect API at start; slow DNS/proxy/TLS can add seconds. - If acceptable on Linux, try

network_mode: "host"

to rule out Docker DNS slowness. - Alternatively, configure a fast DNS or pass

--dns

via

container_create_kwargs

. - Verify latency from inside the image:

time curl -sS <https://api.prefect.cloud/api/health>

(or your API URL). Set via job variables:

job_variables={
  "network_mode": "host",  # if acceptable in your environment
  "env": {
    "PREFECT_TELEMETRY_ENABLED": "0"  # small win, but reduces overhead
  }
}

4) Consider Process worker if isolation is not required - If your primary goal is minimal cold start and you don’t need container isolation, a Process work pool typically begins executing in under a second. Many users run short jobs this way and reserve Docker for heavy/isolated workloads. How to pinpoint the bottleneck (2-minute diagnostic) - Is it Docker start or Python/engine init? - Baseline Docker start:

time docker run --rm your-repo/your-image:tag python -c "print('hi')"

If this takes >2–3s, your image/entrypoint/FS is a big contributor. - If Docker is fast but Prefect run is slow, it’s likely imports/network to the API. - Is it a registry pull/auth check? - Temporarily set

image_pull_policy: Never

. If you suddenly save ~10–20s before “Opening process…”, the pull check was the culprit. - Is it DNS/API latency? - From inside the container, test API latency as above. If slow, try

network_mode: host

or custom DNS. Where to set these in Prefect 3.x - In deployments: pass

job_variables

that match Docker worker config (e.g.,

image

,

image_pull_policy

,

env

,

network_mode

,

volumes

,

stream_output

). - In your Docker work pool: use a base job template so every run gets these defaults. - Create/update via:

prefect work-pool create "docker-pool" --type docker --base-job-template ./base-job-template.json --overwrite

- Then include fields like: ``` { "job_configuration": { "image": "your-repo/your-image:tag", "image_pull_policy": "IfNotPresent", "network_mode": "host", "env": { "PREFECT_TELEMETRY_ENABLED": "0", "PYTHONDONTWRITEBYTECODE": "1" } }

} ``` Useful references - Work pools and workers overview: Work pools and workers - Custom worker concepts (for context on job variables/templates): Developing a custom worker - Worker settings API reference: Worker settings API A few clarifying questions to tailor this further: - Which Prefect version and where is your API (Prefect Cloud or self-hosted)? Any proxy? - Which base image and Python version are you using? Any conda activation in entrypoint? - What is your current

image_pull_policy

? - Are you on Linux, macOS, or Windows/WSL2 (Docker Desktop often adds startup latency)? - Do you set

network_mode

, custom

dns

, or mount any large volumes? If you share your deployment config (or base job template) and Dockerfile, I can suggest targeted changes to shave off those seconds.

Thanks — that helps narrow it down. Given: - Prefect 3.4.7, self‑hosted API -

network_mode=host

and

image_pull_policy=Never

made no difference That strongly suggests the time is not registry or Docker networking related. The two most common remaining culprits are: - Container runtime overhead on the host (e.g., Docker Desktop, filesystem/volume mounts, logging driver) - Python/Prefect import and engine initialization inside your image Fastest way to pinpoint where the time goes Please run these on the same host where the Docker worker runs: 1) Baseline container start cost

time docker run --rm your-repo/your-image:tag python -c "print('ok')"

- If this is >5s (especially 10–20s), the delay is Docker runtime/host related. 2) Python/Prefect import cost inside the image

time docker run --rm your-repo/your-image:tag python -c "import prefect; print('prefect imported')"

3) Your flow module import cost Replace

your_flow_module

with the module your deployment imports at run start.

time docker run --rm your-repo/your-image:tag python -c "import your_flow_module; print('flow imported')"

4) API latency from inside the container Replace URL with your self‑hosted API URL.

time docker run --rm your-repo/your-image:tag python -c "import httpx; print(httpx.get('<https://YOUR_API/api/health>', timeout=5).status_code)"

5) Get more granular Prefect logs during container bootstrap Run your Docker worker with:

PREFECT_LOGGING_LEVEL=DEBUG

Then share the lines around container create/start; the Docker worker logs include timestamps that can show if we’re spending time on create/start/attach vs inside Python. Targeted mitigations depending on what you find A) If “docker run … print('ok')” is already slow - Host/OS: - Docker Desktop (macOS/Windows/WSL2) often adds 5–20s to cold starts. If possible, run your worker on Linux for sub‑second container starts, or use a Process work pool for short flows. - Volumes: - Avoid mounting large directories or network filesystems; even small mounts can add seconds on Desktop. - Logging driver: - Check

docker info | grep -i 'Logging Driver'

. Fluentd/gelf/syslog endpoints that are slow/unavailable can stall start. Prefer

json-file

for lowest friction. - Storage/AV: - Real‑time AV scanning of the Docker data directory (Windows/macOS) can add seconds. Exclude Docker data dirs if allowed. B) If Python imports are slow (step 2/3) - Image/Dockerfile: - Use a slim base (e.g.,

python:3.11-slim

), avoid conda activation in ENTRYPOINT/CMD. - Minimize top‑level imports in your flow module; move heavy imports into tasks/functions. - Multi-stage builds and pruning caches reduce filesystem metadata the interpreter touches at startup. - Add: -

ENV PYTHONDONTWRITEBYTECODE=1

(avoid writing .pyc) -

ENV PYTHONUNBUFFERED=1

- Quick profiling: -

docker run --rm -e PYTHONPROFILEIMPORTTIME=1 your-image python -X importtime -c "import prefect; import your_flow_module"

C) If API handshake is slow (step 4) - Since

network_mode=host

didn’t help, check DNS/resolver or TLS from within the container: - Try using your API IP to rule out DNS. - Ensure any proxy or mTLS is not adding retries. - You can shave very small overhead by disabling telemetry: - Set in job variables or image env:

PREFECT_TELEMETRY_ENABLED=0

D) Prefect worker specifics to try - Disable log attachment overhead: - Set

stream_output: False

in your Docker job variables (logs won’t stream live, but this removes the attach step which can stall on some setups). - Confirm you’re not mounting any unnecessary volumes in the job. - If startup time is paramount and isolation is not, consider a Process work pool for these flows; starts are typically sub‑second. How to set job variables in your deployment ``` flow.from_source(".").deploy( name="my-deploy", work_pool_name="docker-pool", image="your-repo/your-image:tag", job_variables={ "image_pull_policy": "Never",

"network_mode": "host", "stream_output": False, "env": { "PREFECT_TELEMETRY_ENABLED": "0", "PYTHONDONTWRITEBYTECODE": "1", "PYTHONUNBUFFERED": "1", }, }, ) ``` A couple of questions to tailor next steps - What OS is the worker host (Linux vs macOS/Windows/WSL2) and Docker version? - Any volumes mounted by the job template? - What’s your image base and ENTRYPOINT/CMD? Any conda activation or shell wrappers? - What’s the result of the four timing commands above? If you paste your worker logs at DEBUG around container create/start and your Dockerfile/entrypoint, I can help pinpoint and suggest the most impactful changes. Also, if feasible, consider upgrading to the latest 3.4.x to rule out any regressions that have been fixed since 3.4.7.

Great data point — if

docker run --rm your-image python -c "print('hi')"

is ~0.5s, the container itself is fast. That means the 30–40s you’re seeing is almost certainly: - a registry pull/auth check before the container starts, and/or - Python/import + Prefect engine handshake inside the container (loading your flow module + contacting the API) Here’s a targeted checklist to shave that time down quickly. 1) Make sure we’re not pulling - Set the Docker job’s

image_pull_policy

to

IfNotPresent

(or

Never

for testing). A pull/auth handshake can add 10–20s even with a cache hit. - You can set it in the deployment or the work pool’s base job template. Example (during deploy):

flow.from_source(".").deploy(
  name="my-deploy",
  work_pool_name="docker-pool",
  image="your-repo/your-image:tag",
  build=False,
  push=False,
  job_variables={
    "image_pull_policy": "IfNotPresent"
  },
)

2) Profile and reduce import/engine time inside the image Heavy top-level imports in your flow module (pandas, big SDKs, cloud clients) can easily add 5–20s before “set_working_directory”. Measure it inside your image: - Overall Prefect import:

docker run --rm your-image \
  sh -c 'python - <<PY
import time; s=time.time(); import prefect
print("prefect_import_sec", round(time.time()-s, 3))
PY'

- Your flow module import:

docker run --rm -e PYTHONPROFILEIMPORTTIME=1 your-image \
  sh -c 'python -c "import your_flow_module" 2>&1 | tail -n 50'

If these are high: - Move heavy imports inside tasks or functions (lazy import on first use). - Use smaller base images (e.g., python:3.11-slim), and avoid conda activation wrappers. - Consider: -

ENV PYTHONDONTWRITEBYTECODE=1

-

ENV PYTHONUNBUFFERED=1

- Ensure your entrypoint isn’t doing any

pip install

or shell init on start. 3) Check API/DNS latency If imports look fine but you still see a long gap before “set_working_directory”, it can be network to the Prefect API. - Quick test from the image:

docker run --rm your-image sh -c 'time curl -sS ${PREFECT_API_URL:-<https://api.prefect.cloud/api/health>} >/dev/null'

- If this is slow, try: -

job_variables={"network_mode": "host"}

(Linux only) to eliminate Docker DNS overhead - custom DNS via

container_create_kwargs={"dns": ["1.1.1.1","8.8.8.8"]}

- verify proxies/CA settings 4) Turn on worker debug to see where time is spent Start your worker with debug logs to confirm whether it’s pulling or waiting on something:

prefect worker start -p docker-pool --log-level DEBUG

You’ll see messages like “Pulling image …” or container create/start timings. Optional: Process worker if isolation isn’t required If you just need the shortest cold start and can skip container isolation, a Process work pool typically starts executing in under a second. Use Docker only for workloads that need it. If you share: - your deployment’s

job_variables

(or base job template), - the Dockerfile/entrypoint, and - whether you’re on Prefect Cloud or self-hosted (and region/proxy), I can pinpoint the exact culprit. But in most cases like yours, setting

image_pull_policy: IfNotPresent

and moving heavy imports inside tasks cuts startup from ~40s to a few seconds. References - Work pools/workers overview: Work pools and workers - Custom worker/job variables context: Developing a custom worker - Worker settings API (for job variables): Worker settings API

Thanks — that isolates it: your container start is fast (0.5s), Prefect import is fast (0.7s), and the bottleneck is your flow module import (14s). That aligns with the pause you see before “set_working_directory”. Goal: make the entrypoint module that Prefect imports as light as possible so the flow begins quickly. Then import heavy stuff only when the tasks actually run. What to change - Keep your flow entrypoint “thin” - Do not import heavy libraries at module top-level. - Do not construct clients/engines at module top-level (e.g., boto3, bigquery, snowflake, spark, pandas, pyarrow, ML libs). - Avoid reading files/env/credentials at import time. - Lazy-import inside tasks or inside the flow function - Example refactor:

# BEFORE (slow to import)
import pandas as pd
import boto3
from mypkg.heavy import build_model

from prefect import flow, task

@flow
def etl():
    client = boto3.client("s3")
    df = pd.read_csv("<s3://bucket/file.csv>")
    model = build_model()
    ...

# AFTER (fast to import)
from prefect import flow, task

@task
def get_s3_client():
    import boto3
    return boto3.client("s3")

@task
def read_csv(path: str):
    import pandas as pd
    return pd.read_csv(path)

@task
def build_model_task():
    from mypkg.heavy import build_model
    return build_model()

@flow
def etl():
    client = get_s3_client()
    df = read_csv("<s3://bucket/file.csv>")
    model = build_model_task()
    ...

- Split modules so the flow file doesn’t drag in heavy submodules - Put heavy task logic in separate modules and import them only inside task functions or inside the flow function body. - Precompile your project during image build - Install your project as a package (non-editable) so

.pyc

are generated at build time:

# Dockerfile
    FROM python:3.11-slim

    WORKDIR /app
    COPY pyproject.toml poetry.lock* requirements.txt* /app/
    RUN pip install -r requirements.txt  # or poetry/pip-tools

    COPY . /app
    RUN pip install --no-deps .          # install your project (not editable)
    # optional: ensure all pyc are created
    RUN python -m compileall -q /usr/local/lib/python3.11 /app

- This reduces import work at runtime. If you mount your code instead of installing it, you lose this advantage. - Verify the worst offenders with import profiling - You already measured 14s. Pinpoint the exact modules:

docker run --rm -e PYTHONPROFILEIMPORTTIME=1 your-image \
      sh -c 'python -X importtime -c "import your_flow_module" 2>&1 | tail -n 120'

- Move those imports where they’re actually needed. - Optional small wins - Avoid conda activation or shell wrappers in ENTRYPOINT/CMD. - Keep

stream_output: False

in job variables if you can live without live logs; on some setups attach can add seconds (usually minor on Linux). - Don’t set

PYTHONDONTWRITEBYTECODE=1

if you rely on writing pyc at runtime; if you precompile in the image, it’s fine to keep it off or on—measure both. Putting it into your Prefect deployment - Ensure your deployment’s entrypoint module is the “thin” one. - Job variables remain as you have them; they won’t help further because the bottleneck is import time:

job_variables={
    "image_pull_policy": "Never",
    "network_mode": "host",
    "stream_output": False,
  }

If you share the tail of the

-X importtime

output or the list of heavy imports (e.g., pandas, boto3, google-cloud, pyarrow, snowflake-connector, etc.), I can suggest concrete refactors for each.

Thanks for digging into it — 9s to import

prefect.flow

is definitely on the high side. A few notes up-front - In Prefect 3.x, there isn’t a way to use the

@flow

decorator without importing Prefect at some point. The best you can do is delay when you pay that import cost or reduce it. - Import time can be dominated by environment factors (slow/remote filesystems, lots of installed packages/entry-points) as much as Prefect itself. Can you share a quick import profile? This will tell us exactly where the time is going:

python -X importtime -c "from prefect import flow"

If you can paste the summary (or run it through tuna), we can target the hotspots. Practical mitigations you can try now - Defer imports to when you actually run the flow - If your module does a lot of non-Prefect work on start-up, move the Prefect imports inside functions or behind the CLI entrypoint so you only import Prefect on the code path that runs flows:

def get_flow():
      from prefect import flow, task

      @task
      def t():
          ...

      @flow
      def f():
          t()
      return f

  if __name__ == "__main__":
      get_flow()()

- For type hints only, avoid runtime imports:

from typing import TYPE_CHECKING
  if TYPE_CHECKING:
      from prefect import Flow

- Keep a long-lived process “warm” instead of cold-starting repeatedly - If you’re invoking flows frequently, run them via a Prefect worker/process that stays up; it imports Prefect once then executes many flow runs. That amortizes import time. - Trim your environment - Prefect will discover available integrations/plugins via Python entry points. If your env has hundreds/thousands of packages, that scan can be slow. - Use a lean virtualenv just for your flows. Avoid installing heavy optional extras you don’t need (e.g., not

prefect[all]

). - Ensure site-packages is on local disk, not a network/NFS/slow container volume. - Keep Prefect up to date - We do periodically improve startup behavior. If you’re not on the latest 3.x, upgrading may help. Why imports can be slow - Entry-point/plugin discovery: scanning installed distributions can be expensive in large environments. - Pydantic v2 and other foundational libs are non-trivial to import. - Logging/config/settings initialization. - Filesystem latency (containers, networked filesystems) can amplify all of the above. References - Prefect plugins API (entry point loading and discovery): prefect.plugins - Import utilities (lazy import helpers Prefect uses internally): prefect.utilities.importtools If you can share: - Python version and OS - Prefect version - Whether you’re in a container/WSL/remote FS - Rough count of installed packages (

pip list | wc -l

) - The

-X importtime

summary …I can suggest more targeted fixes. If this is reproducibly 9s in a lean env, I’m happy to raise a performance issue for us to investigate.

Thanks for the details — that helps. Given Debian 11 + Python 3.11 + container + ~255 packages, a big contributor is typically entry-point/distribution scanning and filesystem latency during import. In Prefect 3.4.7 there isn’t a supported env var to “turn off” plugin discovery globally, so the best wins are usually environment and import-structure changes. Concrete things you can try (in order of impact) - Use a lean virtualenv in the container - Many packages on sys.path = slower importlib.metadata scans. In your Dockerfile, create a venv and only install what you need.

FROM python:3.11-slim
  RUN python -m venv /venv
  ENV PATH="/venv/bin:$PATH" PYTHONDONTWRITEBYTECODE=0
  COPY requirements.txt .
  RUN pip install --upgrade pip && pip install --no-cache-dir -r requirements.txt

- If you currently install OS Python packages via apt or are not using a venv, moving to a minimal venv often cuts import time substantially. - Pre-warm .pyc and Prefect at image build time - Ensure

.pyc

files can be written (avoid

PYTHONDONTWRITEBYTECODE=1

). - During the image build, import Prefect once to precompile bytecode:

RUN python - <<'PY'
  import prefect  # triggers compilation of many modules to .pyc in the venv
  print("prefect pre-imported")
  PY

- This avoids repeated on-first-import compilation cost at runtime, which is amplified on container filesystems. - Keep a long-lived process warm - If you’re cold-starting the container to run a single flow, you’ll pay the import cost every time. Running flows via a persistent Prefect worker/process amortizes the import cost across many runs. - Minimize installed integrations/plugins - Uninstall Prefect collections/integrations you do not use (e.g.,

prefect-aws

,

prefect-docker

,

prefect-kubernetes

, etc.). Each can register entry points that add to discovery time. - Trim general packages too; 255 is on the higher side for fast cold starts. A “flows-only” venv often lands well under 100. - Adjust your import structure to defer when you pay the cost - Import Prefect only on the path that actually runs your flow:

def get_flow():
      from prefect import flow, task

      @task
      def t():
          ...

      @flow
      def f():
          t()
      return f

  if __name__ == "__main__":
      get_flow()()

- For type checking, guard imports:

from typing import TYPE_CHECKING
  if TYPE_CHECKING:
      from prefect import Flow

- Also try importing the decorator from the submodule and measure if it’s faster in your env:

# Compare these two in your container; in some envs submodule import can be a bit lighter
  from prefect import flow
  # vs
  from prefect.flows import flow

- Sanity-check entry point overhead in your env

python -c "import time, importlib.metadata as im; t=time.perf_counter(); eps=list(im.entry_points()); print('all entry points:', len(eps), 'scan:', time.perf_counter()-t)"
  python -c "import time, importlib.metadata as im; t=time.perf_counter(); eps=im.entry_points().select(group='prefect.plugins'); print('prefect plugins:', len(tuple(eps)), 'scan:', time.perf_counter()-t)"

If these are slow or return large counts, slimming the venv will help. If you’re open to a quick baseline comparison, try a minimal test container:

docker run --rm -it python:3.11-slim bash -lc "pip install -q prefect==3.4.7 && python -X importtime -c 'from prefect import flow' 2>&1 | tail -n 40"

If that’s much faster than ~9s, the difference is likely your env size/filesystem. If it’s still slow, we can dig deeper and I can file a performance issue. Related docs: - Plugins/entry points overview: prefect.plugins - Import helpers used internally (context on lazy imports): prefect.utilities.importtools If you can share:

- Whether you’re using a venv inside the container - Whether

PYTHONDONTWRITEBYTECODE

is set - Whether the container is cold-started per run …I can narrow this further or provide a tuned Dockerfile.