Got it. Assuming you’re on Prefect 3.x: the Process worker runs each flow run in a fresh subprocess by design. There isn’t a setting to reuse a single Python process across multiple flow runs. This isolation is intentional (clean environment, fewer side-effects), but it does mean a new process per run. If your goal is to avoid the overhead of per-run process creation, here are workable patterns: - Use a long-running “service” flow that processes many units of work in one run - One flow run, one process; loop inside the flow to handle multiple items/jobs. - You still get Prefect logging/state on that single long-lived run.

from time import sleep
from prefect import flow, task

@task
def handle(item):
    # do work
    return f"processed {item}"

def poll_queue():
    # replace with your queue (SQS, Redis, DB, etc.)
    return None

@flow
def service_flow(poll_interval: int = 5):
    while True:
        item = poll_queue()
        if item is None:
            sleep(poll_interval)
            continue
        handle(item)

- Use a “driver” flow that calls other flows as subflows - Calling another

@flow

function from within a running flow executes inline in the same OS process as the parent (while still recording a separate subflow run in the UI). - This lets you keep separate flow logic but avoid spawning a new process for each subflow call.

from prefect import flow, task

@task
def compute(x: int) -> int:
    return x * 2

@flow
def child_flow(x: int) -> int:
    return compute(x)

@flow
def driver_flow(items: list[int]) -> list[int]:
    results = []
    for x in items:
        # Executes inline in the same process as the driver flow
        results.append(child_flow(x))
    return results

- Batch more work per run - If many short flows are the issue, combine units of work into a single run (e.g., pass a list of jobs as a parameter, or have the flow pull N items per iteration). - Cap concurrency to reduce churn - While this won’t remove per-run processes, setting a deployment/work-pool concurrency limit can prevent a surge of subprocesses from spinning up at once. If you can share a bit more about your objective (e.g., reducing cold-start time for heavy libraries, reusing a database connection, GPU warmup, etc.), I can suggest a more targeted pattern and help adapt your current deployments.

Thanks — that context helps. Short version: with the Process worker in Prefect 3.x you can’t reuse a single Python process across runs, so to reduce churn/cold-start and keep slab reclaimable from climbing, the levers that work are about: - reducing how many processes you start - making each process live longer and do more work - cutting per-run filesystem and stdout churn Here are the most effective, actionable steps: 1) Cap concurrent subprocesses - Set a work pool concurrency limit so you don’t start too many processes at once.

prefect work-pool update "my-process-pool" --concurrency-limit 4

- Optionally also set a per-deployment concurrency limit so a busy deployment doesn’t flood the pool:

python
from prefect import flow

@flow
def my_flow():
    ...

my_flow.to_deployment(
    name="prod",
    work_pool_name="my-process-pool",
    concurrency_limit=2,  # per-deployment cap
)

2) Switch many short runs to a long-running “service” flow - One flow run = one OS process. If you currently trigger lots of tiny runs, replace them with a single long-lived flow that loops and pulls work from your queue/database, optionally parallelizing tasks inside the same process with a task runner.

from time import sleep
from prefect import flow, task
from prefect.task_runners import ConcurrentTaskRunner

@task
def handle(item):
    # heavy libs imported once at module import; reused across loop
    return f"processed {item}"

def poll_batch(n=50):
    # fetch up to n items from your queue/db
    return []

@flow(task_runner=ConcurrentTaskRunner())
def service_flow(poll_interval=2):
    while True:
        batch = poll_batch(n=50)
        if not batch:
            sleep(poll_interval)
            continue
        # process in-parallel threads inside the SAME process
        _ = [handle.submit(x) for x in batch]

- If you want to keep separate flows, you can call them as subflows from this driver — subflows execute in the same OS process as the parent in Prefect 3, which avoids new process spawns per unit of work. 3) Reuse a stable working directory and cut stdout streaming - By default the Process worker uses a fresh temporary directory per run. That can create filesystem churn (and dentry/inode slab growth). Point runs at a stable working directory and turn off output streaming. - Update your work pool’s base job template:

json
{
  "job_configuration": {
    "command": "{{ command }}",
    "env": "{{ env }}",
    "labels": "{{ labels }}",
    "name": "{{ name }}",
    "stream_output": "{{ stream_output }}",
    "working_dir": "{{ working_dir }}"
  },
  "variables": {
    "properties": {
      "stream_output": { "type": "boolean", "default": false },
      "working_dir": { "anyOf": [{"type":"string","format":"path"},{"type":"null"}], "default": "/var/lib/prefect/flows" },
      "env": { "type": "object", "additionalProperties": {"anyOf":[{"type":"string"},{"type":"null"}]}, "default": { "TMPDIR": "/dev/shm/prefect-tmp" } },
      "command": {"anyOf":[{"type":"string"},{"type":"null"}], "default": null},
      "name": {"anyOf":[{"type":"string"},{"type":"null"}], "default": null},
      "labels": {"type":"object", "additionalProperties":{"type":"string"}, "default": {}}
    },
    "type": "object"
  }
}

Apply it:

prefect work-pool update "my-process-pool" --base-job-template ./base-job-template.json

Tips: - Set working_dir to a persistent path to avoid per-run temp dirs. - Put TMPDIR on tmpfs (e.g., /dev/shm/prefect-tmp) to reduce slab growth from short-lived files. - Set stream_output to false to reduce stdout/stderr churn. 4) Batch more work per run - If you must keep scheduled runs, process a batch (e.g., 100–1000 items) per run instead of 1 item/run; or have the flow pull N items until a time/record budget is met. 5) Keep imports and resources warm inside one process

- In the service/driver flow pattern, heavy libraries and connections are created once and reused across iterations. For heavy GPUs/ML or large DB drivers, this typically yields the biggest cold start win. 6) Right-size worker count - Prefer fewer workers with controlled concurrency over many workers each starting processes. Start 1 worker per node with a modest pool concurrency limit (e.g., equals physical cores or lower if I/O bound). If slab reclaimable is still climbing: - Verify the per-run working dirs and tmp files aren’t being created/deleted wildly (the stable working_dir + tmpfs TMPDIR usually fixes this). - Consider rotating or lowering log verbosity (e.g., set LOG_LEVEL / PREFECT_LOGGING_LEVEL to WARNING) to reduce log buffers. - As a last resort, periodically restart the worker off-hours to force slab reclaim, but most teams don’t need this once the above changes are in place. If you can share: - average run duration - current concurrency (pool and deployment) - size of the batches you process - whether you’re using lots of short-lived temp files I can help tune specific numbers and provide a template aligned with your environment.

Great questions. - Is a long-running “service” flow a problem? - It’s a supported pattern in Prefect 3.x and commonly used to reduce cold starts. One OS process runs a single flow run for a long time and loops to handle work. - Caveats/best practices: - Don’t set a small

timeout_seconds

on the flow; leave it unset (default None). - Use modest sleep/poll intervals so cancellation/shutdown is responsive. - Watch memory growth; reuse clients/connections and periodically recycle any large caches. - Add an automation to restart the service flow if it exits in a failed/crashed state. - Keep log volume reasonable (e.g., raise log level or throttle prints) to reduce I/O churn. - Prefer a stable working directory and tmpfs TMPDIR as discussed earlier to reduce slab growth. - How to run multiple service flows in the worker - Create one deployment per service flow and point them all at the same Process work pool. - Give each service deployment a concurrency_limit=1 so only one instance runs at a time. - Ensure the work pool’s concurrency limit is high enough to run all services concurrently (these runs will stay active and occupy slots). - Start your Process worker and then trigger each service deployment once (they’ll keep running). Example:

from prefect import flow

@flow(log_prints=True)
def service_a():
    while True:
        # poll external source A
        # process batch
        ...

@flow(log_prints=True)
def service_b():
    while True:
        # poll external source B
        # process batch
        ...

# Create deployments with single-instance concurrency
service_a.to_deployment(
    name="svc-a",
    work_pool_name="my-process-pool",
    concurrency_limit=1,  # ensures one instance max
)
service_b.to_deployment(
    name="svc-b",
    work_pool_name="my-process-pool",
    concurrency_limit=1,
)

- Start the worker (ensure pool exists and has a concurrency limit >= number of services you want running):

prefect work-pool update "my-process-pool" --concurrency-limit 4
prefect worker start --pool "my-process-pool"

- Trigger each service once (manual kick-off):

prefect deployments run "service_a/svc-a"
prefect deployments run "service_b/svc-b"

For durability, create an automation that listens for “Flow run failed or crashed” on each service deployment and triggers a new run so the service self-recovers. - Can these service flows poll for Prefect flows? - Yes, you have two options: 1) Trigger other deployments via the API (creates new Prefect flow runs handled by workers; note this spawns a new process per run):

from prefect.deployments import run_deployment

    @flow
    def service_driver():
        # when condition is met, create a new flow run of another deployment
        run_deployment(
            name="other-flow/production",
            parameters={"x": 1},
            # as_subflow defaults to True; set to False if you want it fully independent
            as_subflow=False
        )

2) Call flows as subflows inline (no new process; runs inside the same OS process as the service flow):

from prefect import flow, task

    @task
    def work(x): return x*2

    @flow
    def child_flow(x): return work(x)

    @flow
    def service_driver_inline(batch: list[int]):
        # executes child flows inline in the same process – lowest churn
        results = [child_flow(x) for x in batch]
        return results

- If your goal is to minimize churn/cold starts, prefer calling flows inline as subflows. Use

run_deployment

only when you specifically need separate deployment isolation/routing through a work pool. Notes to keep slab reclaimable in check with long-running services: - Use a stable working_dir and disable stream_output via the work pool’s base job template so runs don’t create/tear down temp dirs or produce high-volume stdout.

- Point TMPDIR to tmpfs (e.g.,

/dev/shm/prefect-tmp

) via job variables env to keep short-lived files out of disk-backed caches. - Keep work pool and deployment concurrency limits reasonable so you don’t start more long-lived processes than the node can comfortably host. If you’d like, share: - how many services you plan to run concurrently, - average per-batch work size and poll interval, - whether services will trigger downstream flows frequently, and I’ll help tune pool/deployment concurrency and provide a ready-to-apply base job template for your worker.