<@ULVA73B9P>, I have a self hosted prefect server ...
# ask-marvin
t
@Marvin, I have a self hosted prefect server and worker using prefect helm chart, where will my flow code or deployment code reside to be registered as a flow or deployment
m
thought for 125.4 seconds
Great question. Short answer: your Prefect Server does not store your flow code. It only stores deployment metadata (name, entrypoint, parameters, schedules, where to fetch code, which work pool to use, etc.). With a self‑hosted server and a Kubernetes worker (via the Helm chart), your code lives wherever you choose (Git, container image, object storage, or a mounted volume) and the worker fetches it at run time based on the deployment configuration. Common patterns for where your code “resides” - Git repository (recommended): Store your code in a Git repo. Your deployment points to the repo/entrypoint and the worker clones it at runtime. - Docker image: Bake your code into a container image and reference that image in the deployment. The Kubernetes job runs that image directly. - Remote storage (S3/GCS/Azure): Upload code to object storage and configure the deployment’s storage so the worker downloads it at runtime. - Mounted volumes (K8s PV/PVC): Less common; you mount a shared volume into the pod and reference a local path. What the server actually stores - Deployment metadata only: flow name, entrypoint like
path/to/file.py:flow_func
, work pool/work queue, schedule, parameters, and the code “source” info (e.g., Git URL + ref or image tag). - No flow files are stored in the server DB; workers fetch code when a run starts. How to “register” your deployments - Programmatic (Prefect 3.x) - Point a flow to your code source, then deploy:
Copy code
from prefect import flow

  @flow
  def my_flow():
      print("hello from k8s")

  # Run this once from your laptop/CI to register the deployment
  (
      my_flow
      .from_source(
          source="<https://github.com/acme/data-pipelines>",
          entrypoint="flows/etl.py:my_flow",
      )
      .deploy(
          name="etl-prod",
          work_pool_name="k8s-pool",
          cron="0 * * * *",
      )
  )
- The server records a deployment pointing to your Git repo and entrypoint. When scheduled/triggered, the K8s worker clones the repo, imports
my_flow
, and runs it. - CLI (Prefect 3.x) - Initialize a project, commit
prefect.yaml
, and deploy from the repo root:
Copy code
prefect deploy ./flows/etl.py:my_flow --name etl-prod
- The CLI uses
prefect.yaml
to define how the worker should pull code (e.g., clone a Git repo or use an image). If you don’t pass a pool, you’ll be prompted. See
prefect deploy --help
for options. Kubernetes specifics with the Helm worker - Git-based deployments: the worker’s job clones your repo into the pod at start. Provide credentials via blocks or environment variables as needed for private repos. - Image-based deployments: set an
image
when deploying; the K8s job runs that image. This is the most hermetic option. - Remote storage: configure your deployment’s storage (S3/GCS/Azure). The job will download the code before execution. Docs and references - How to store and fetch flow code: Store flow code - Kubernetes worker/how it runs: Kubernetes deployments - Workers (vs agents) overview: Upgrade to workers If you share how you prefer to distribute code (Git vs image vs object storage, and whether your repo is private), I can give you a concrete
prefect.yaml
or deployment snippet tailored to your setup.
t
@Marvin can you share some video lecture links around prefect
m
thought for 79.3 seconds
Absolutely! Here are solid starting points for Prefect video content (focused on Prefect 3.x): - Official YouTube channel: Prefect on YouTube - Channel playlists: All Prefect playlists - Prefect 3.x content (filtered search on the channel): - “Prefect 3” - “deployments” - “workers” - “kubernetes” - “automations” More official resources with recorded sessions - Community hub (events, webinars, office hours): Prefect Community - Events and webinars archive: Prefect Events - Summit talks (great deep dives; look for 2024 recordings): Prefect Summit overview If you tell me what you want to learn (e.g., “Kubernetes deployments,” “migrating to 3.x,” “schedules/automations,” or “best practices”), I’ll pull a short list of specific videos tailored to that.
t
@Marvin we use serve and deploy to register deployments from local, how will a flow code located in git be registered as a deployment ?
m
thought for 146.5 seconds
In Prefect 3.x, you register a deployment that points to Git by creating the deployment “from source” (the Git repo) instead of from local files. The server stores the repo URL + entrypoint and your Kubernetes worker clones the repo at runtime. Two common ways to do it 1) Programmatically (explicit Git source)
Copy code
from prefect import flow
from prefect.storage import GitRepository  # GitCredentials optional for private repos

@flow
def my_flow():
    ...

repo = GitRepository(
    url="<https://github.com/acme/data-pipelines.git>",
    branch="main",              # or commit_sha="abc123"
    # credentials=GitCredentials(...)  # for private repos
)

(
    my_flow
    .from_source(source=repo, entrypoint="flows/etl.py:my_flow")
    .deploy(
        name="etl-prod",
        work_pool_name="k8s-pool",
        cron="0 * * * *",
    )
)
2) CLI from a local clone of the Git repo (auto-detects Git) - Run the command from inside your repo so Prefect can detect the origin URL and ref:
Copy code
prefect deploy ./flows/etl.py:my_flow --name etl-prod --pool k8s-pool
- Prefect will record the repo URL and current branch/commit; your worker will clone that repo when the run starts. Notes -
serve
is great for local/ephemeral development. For durable runs on your K8s worker, use
.deploy(...)
or
prefect deploy
. - For private repos, provide credentials via a GitCredentials block, or use standard Git auth in your worker pod (e.g., K8s secret + SSH key or HTTPS PAT). Avoid embedding tokens directly in URLs when possible. Docs - Store flow code in Git and run from workers: Store flow code If you share whether your repo is public/private and which provider (GitHub/GitLab/Bitbucket) you’re using, I can show a concrete auth setup for your K8s worker.