<@ULVA73B9P> Cuurrently we have a scheduler worker...
# ask-marvin
s
@Marvin Cuurrently we have a scheduler worker that receive event from different source and then create a deployment for a flow based on the event. We also use a work pool in K8s to spawn worker to run task, the problem is we dont want to install all the libraries for running all the flow inside scheduler worker, is there any solution to solve this?
m
thought for 307.6 seconds
Short answer: yes—keep your scheduler “thin” and move all flow dependencies into the Kubernetes pods that actually run the work. In Prefect 3.x you can do this by: - Creating deployments that pull code from a remote source (e.g., Git) and run in per-deployment Docker images - Having your scheduler only trigger deployments (no importing flow code, no heavy libs installed) Here’s a recommended setup. 1) Package dependencies in per-flow images - Build a Docker image per flow (or per group of flows) with all needed Python libs. - Your Kubernetes work pool will launch pods with the image you specify on each deployment. 2) Create deployments from remote source (no local imports on the scheduler) - Point deployments to your repo and specify the image and work pool. This keeps the scheduler free of flow dependencies. - Example (run this in CI/CD or a thin admin box that just has Prefect installed):
Copy code
from prefect import flow
from prefect.runner.storage import GitRepository
# If the repo is private, also import GitCredentials

etl = flow.from_source(
    source=GitRepository(
        url="<https://github.com/acme/data-flows.git>",
        # credentials=GitCredentials(username="...", access_token="..."),
        branch="main",
    ),
    entrypoint="flows/etl.py:etl",
)

etl.deploy(
    name="etl-prod",
    work_pool_name="k8s-pool",
    image="<http://ghcr.io/acme/etl:1.2.3|ghcr.io/acme/etl:1.2.3>",
    job_variables={
        "namespace": "prefect",
        "image_pull_policy": "IfNotPresent",
        # add other k8s vars as needed: env, service_account_name, image_pull_secrets, etc.
    },
    # For prebuilt images: build=False, push=False
)
Docs: - Flow code storage with from_source - Job variables (per-deployment K8s settings) - Kubernetes work pools 3) Have your scheduler only trigger deployments - Your scheduler receives events and calls `run_deployment`—no need to import flow code or install its libs.
Copy code
from prefect.deployments import run_deployment

# Trigger an existing deployment by its "flow-name/deployment-name"
run = run_deployment(
    name="etl/etl-prod",
    parameters={"date": "2025-11-07"},
    # Optional: override the image at trigger-time if you must
    # job_variables={"image": "<http://ghcr.io/acme/etl:1.2.4|ghcr.io/acme/etl:1.2.4>"}
)
- Alternatively, use the Prefect REST API or client to create a flow run from a deployment (still no imports in your scheduler). Docs: Run deployments programmatically API ref: Create flow run from deployment (REST) Notes and gotchas - Avoid removed 2.x patterns in 3.x: don’t use
Deployment.build_from_flow()
or `prefect deployment build`; use
flow.from_source(...).deploy(...)
or
prefect deploy
. - If you must create deployments dynamically on events, you can still do so from your scheduler as long as you: - Reference flows via
from_source(...)
(so you don’t import code locally) - Set
image=...
(and, if the image is prebuilt,
build=False, push=False
) - For private repos, use
GitCredentials
in
GitRepository
. Questions to tailor the best approach for you: - Do you want to create deployments on-the-fly per event, or just trigger pre-created deployments? - Where is your flow code hosted (GitHub/GitLab, private repo)? - Do you already have per-flow Docker images, or should we outline a simple CI step to build/push them?