The common practice is: Use

.serve()

when you want an always-running process that can start runs quickly, but know that it is not a truly “preloaded Python execution environment” for each run. With

flow.serve()

in Prefect 3.x: - the runner process stays alive - but each flow run is executed in a new subprocess - so Python modules are imported again for each run So if your goal is: “I do not want infra to spin up after the deployment is triggered” then

.serve()

is a good fit. If your goal is: “I want the exact Python interpreter with all libraries already loaded in memory and reused run-to-run” then

.serve()

does not do that. ### Community practice Usually people choose one of these: 1. Run

.serve()

in a long-lived machine/container - best when you want low trigger-to-start latency - no pod/container/job creation on every run - common for “always warm” execution 2. Use a worker + process work pool - also good for low overhead on a static machine - still not shared in-memory execution per run 3. Use Docker/Kubernetes work pools with prebuilt images - best for reproducibility/scaling - but there is still container/pod startup overhead ### Best match for your requirement If you want execution to be available immediately, the usual approach is: - install all dependencies ahead of time - start

flow.serve()

on a VM / container / service that stays running - keep that process alive 24/7 Example:

python
from prefect import flow

@flow
def my_flow():
    print("ready to run")

if __name__ == "__main__":
    my_flow.serve(
        name="always-on-flow",
        pause_on_shutdown=False
    )

Then run that file in an always-on environment: - VM - Docker container - ECS service - Kubernetes Deployment - systemd service That gives you a warm runner, but not a warm in-memory flow subprocess. ### Important limitation If your imports are heavy,

.serve()

will still pay import cost per run because Prefect launches flow runs in subprocesses. So to reduce latency further, the practical options are: - make imports lighter - move heavy initialization into an external service - keep data/models cached outside the flow process - use a prebuilt container image with all libraries installed already ### Docs Relevant Prefect docs: - Serve a flow - Serve flows in Docker - Work pools - Manage work pools - Docker deployments - Kubernetes deployments ### Bottom line If you want no infrastructure cold start, use

.serve()

in a long-running environment. If you want no Python import/loading cost per run, Prefect does not really provide that model out of the box with

.serve()

because runs happen in subprocesses. If you want, I can show you the best architecture for your case, for example: - always-on VM +

.serve()

- Dockerized

.serve()

- worker/process pool - Kubernetes with lowest startup latency

Yes — that’s exactly the limitation, and your reading is correct. If re-importing heavy libraries in a new subprocess per run is unacceptable, then the honest answer is: the community usually does not solve that inside Prefect’s normal execution model. Instead, they move the hot state outside Prefect. ### What people typically do instead #### 1. Keep the heavy runtime in a separate long-lived service This is the most common workaround. Pattern: - Prefect orchestrates - a long-running app/service keeps models/libraries/data loaded in memory - the flow just sends work to that service Examples: - FastAPI service - gRPC service - model server - internal worker daemon - queue consumer So Prefect becomes the scheduler/orchestrator, not the place where the heavy in-memory runtime lives. Example shape:

python
from prefect import flow
import requests

@flow
def score_batch(payload: dict):
    response = <http://requests.post|requests.post>("<http://my-hot-service:8000/run>", json=payload)
    response.raise_for_status()
    return response.json()

This is usually the cleanest answer if you need: - loaded ML models - huge Spark/session init - expensive imports - in-memory caches - GPU contexts kept warm #### 2. Use Prefect to trigger a job queue, not do the heavy work directly Another common pattern: - Prefect flow writes a message to Redis / SQS / Kafka / RabbitMQ - a long-lived consumer process handles the job with libraries already loaded That consumer can keep: - models in RAM - DB pools open - client sessions alive - large shared state warm #### 3. Wrap the heavy code in a permanently running actor/worker process Instead of having Prefect execute the heavy logic directly, Prefect calls into a resident process you control. For example: - a custom Python daemon - Ray actor - Celery worker - Dask worker with long-lived state - custom multiprocessing service Prefect then coordinates around it. ### What the community generally does not do They generally do not use Prefect itself as a “hot interpreter reuse” runtime. That’s because Prefect is built around: - isolated runs - reproducible execution - observable run boundaries - safer failure handling Reusing the same in-memory interpreter across runs would trade off a lot of that isolation. ### Why this exists It feels odd, but it is intentional. Fresh subprocesses help avoid: - memory leaks carrying across runs - mutated globals - broken state after one failed run - dependency/runtime contamination between executions So the limitation is real, and it’s not usually “worked around” within Prefect. ### Practical recommendation If your libraries are very expensive to load, the strongest community-style architecture is: Prefect for orchestration + separate always-hot execution service for the expensive runtime. For example: - Prefect schedules work - Prefect handles retries, parameters, dependencies, SLAs - hot service performs the actual heavy computation ### When to still use Prefect tasks/flows directly Only if: - imports are acceptable - startup cost is small enough - isolation is more important than warm-memory reuse ### Bottom line If “libraries must already be loaded in memory and reused between executions” is a hard requirement, then don’t run that code as the Prefect flow execution process itself. Use Prefect to invoke a long-lived service/process that owns the hot runtime. If you want, I can sketch 3 concrete architectures for this: 1. Prefect + FastAPI hot service 2. Prefect + Redis queue + warm worker 3. Prefect + Ray/Celery/Dask long-lived workers and show which one fits your workload best.

Yes — Background Task Workers are the Prefect feature that is much closer to what you want. ### Short answer - Prefect Background Task Workers: yes, they can keep a hot Python process alive and reuse loaded libraries between task executions. - Celery with Prefect 3.x: not an official Prefect execution backend in the normal deployment/worker model. ### Important distinction This applies to tasks, not standard flow-run execution via deployments /

.serve()

/ workers. So if your requirement is: “I need a long-lived Python worker process with libraries already loaded, and I want executions to reuse that process” then Background Task Workers are the native Prefect option that best matches that. ### How Background Task Workers help Background task workers are long-lived worker processes that execute task runs in-process: - no fresh subprocess for each execution - imports can stay loaded - in-memory state can persist - good for “hot worker” patterns This is why the docs position them as something you can use instead of Celery-style task queues. Docs: - Run background tasks - Background tasks guide - Background tasks concept ### But there’s a catch Background task workers are for task queue style execution, not a drop-in replacement for every flow deployment pattern. That means you may need to redesign slightly: - use Prefect flows as orchestration - offload the heavy work into background tasks - let the long-lived task worker own the hot libraries/models/connections ### Example architecture

python
from prefect import flow, task

@task
def heavy_compute(x):
    import pandas as pd
    import numpy as np
    return x * 2

@flow
def orchestrate():
    future = heavy_compute.delay(10)
    return future

Then a background task worker serves that task in a long-lived process. ### About Celery Prefect 3.x does not have a standard official

prefect-celery

execution backend like older “Prefect + task runner backend” mental models people may expect. So if you use Celery, it is usually as: - a separate system alongside Prefect - your own integration pattern - not the main officially recommended Prefect 3 deployment runtime In practice: - If you already have Celery and want Prefect only for orchestration, you can have a Prefect flow call Celery jobs. - But if your goal is specifically hot task workers with loaded Python libs, Prefect Background Task Workers are the more native option. ### Community guidance in practice If you want hot in-memory execution: - use Background Task Workers If you want standard flow deployments: - use

.serve()

or work pools, but accept subprocess/container startup behavior If you already run a mature Celery system: - Prefect can orchestrate around it - but Prefect itself won’t turn normal deployments into Celery-style hot workers ### Best fit for your requirement Based on what you said, I would strongly suggest: refactor the heavy part into background tasks served by Prefect Background Task Workers That is the closest native Prefect answer to: - no re-import each execution - libraries already loaded - immediately available execution If you want, I can give you a concrete migration example from: 1.

flow.serve()

to 2.

flow

+

background task worker

so you can see exactly how to restructure it.

Yes — for the task, not for the flow run itself. ### Short answer You can do this: -

flow.serve()

for the orchestration layer -

task.delay(...)

for the heavy work - a long-lived background task worker executes that delayed task without creating a new subprocess for the task execution But: - the flow run started by

.serve()

still runs in its own subprocess - only the

.delay(...)

task can run on the long-lived task worker ### So the model is 1.

.serve()

receives a flow run 2. Prefect starts a subprocess for that flow run 3. inside that flow, you call

my_task.delay(...)

4. that submits the task run to Prefect 5. a Background Task Worker picks it up 6. the task executes in the worker’s long-lived process So you cannot make

.serve()

itself run the whole flow in the already-hot servant process. You can make the heavy part happen in the hot task worker. ### Practical implication If your expensive imports are inside the task worker process, then this can absolutely reduce the pain: - flow subprocess stays light - heavy libraries are loaded in the task worker once - repeated delayed tasks reuse that loaded environment ### Pattern to use

python
from prefect import flow, task

@task
def heavy_task(x):
    import pandas as pd
    import numpy as np
    return x * 2

@flow
def orchestrator(x: int = 10):
    future = heavy_task.delay(x)
    return str(future.task_run_id)

Then separately run a background task worker serving

heavy_task

. ### Important limitation If your flow needs to directly execute code in the same hot process, this won’t happen with

.serve()

. Only delayed tasks can be offloaded to the long-lived task worker. ### Architectural takeaway If you want to use Prefect this way, the usual pattern is: - keep the flow thin - move heavy logic into delayed background tasks - let the background task worker own the preloaded libraries/models/state ### When this works well Good fit: - ML inference tasks - expensive imports - data processing libraries - long-lived clients/connections - repeated background units of work Less good fit: - if the entire flow body itself must be hot and reused in-memory - if orchestration logic also depends on heavy state ### Relevant docs - Run background tasks - Background tasks - Serve a flow ### Bottom line Yes, you can combine

.serve()

and

.delay(...)

so that the delayed task uses the long-lived worker process. No, you cannot make

.serve()

itself avoid the flow subprocess model. If you want, I can show you a full working example with: - one file for the served flow - one file for the background task worker - how to wait for / retrieve delayed task results cleanly