@Marvin if i deploy my flows to run on with kubernetes workers, how can i configure it so the k8 workers pull the flow from a github repo instead of having to bake the code into the image, can i use

from_source

? if so how would that work?

process_content.deploy(
        work_pool_name="k8s-cpu",
        image="my-image",
        build=False,
        push=False,
        name="test"
    )

Configuring Kubernetes workers to pull flows from a GitHub repository rather than baking the flow code directly into the Docker image involves a few steps. Sadly, the documents provided don't outline the specific steps to use

from_source

in a Kubernetes context or explain how to configure Kubernetes workers to pull code from a repository. However, assuming you want to avoid baking your flows into the image, you would typically use some component of your workflow to clone the repository at runtime. This isn't straightforward and typically requires customizing either your Docker image or the Kubernetes job definition to handle the code fetching process. Instead of cloning the entire repo at runtime, a more common and more manageable approach would be using the Prefect Storage interface to pull specific flows from your repository. With Prefect 2.x, there might be new ways to configure this that I'm not aware of yet as I have no direct experience with it. If you've stumbled upon the

from_source

function and believe it's the solution, then it would likely be a method of the

Flow

class or similar that takes a path to a Python file or module and loads the flow from it. You might configure it in such a way that it dynamically pulls the latest flow code from your repository when the flow is run on a Kubernetes worker. You would still need to handle credentials and access to your private repository for this to work, of course. For more accurate and detailed information on configuring such a setup, I would recommend reviewing the Prefect 2.x documentation or reaching out to the Prefect community or support directly for guidance tailored to your specific needs and the features available in the latest version of Prefect.

yeah you can use

from_source

, which will essentially pull the code at runtime on whatever infra your work pool says alternatively, you can be more explicit and define your deployment in a

prefect.yaml

with a

git_clone

pull step and your k8s work pool @Marvin send some links on that last part

To use

from_source

in Prefect deployments, you can define your flow to be pulled from a remote location at runtime. Unfortunately, the provided excerpts don't contain the complete information or step-by-step examples. For defining a deployment in

prefect.yaml

with a

git_clone

pull step and a Kubernetes work pool, you should create a YAML file that Prefect uses during the deployment process, including Kubernetes-specific configuration. The excerpts did not provide the exact steps for using the

git_clone

step within this file. However, the documentation suggests that you can define Kubernetes objects and include build and push steps for Docker images within the

prefect.yaml

. Here are links to the relevant documentation that can guide you further: - For general information on Prefect deployments and using `from_source`: Prefect Deployment Guide - For information on defining deployments using

prefect.yaml

with Kubernetes: Prefect Kubernetes Guide Due to the nature of documentation and features evolving over time, the specifics might differ slightly. Please refer to these resources and ensure they align with the version of Prefect you're using.

Thanks both! Is that as simple as just adding a .from_source before the .deploy? Will that pull only the flow code, or will it pull the entire repo (as the flow code uses other utilities I would also want to be pulled from the repo)? We’ve preferred using the Python .deploy approach over using the prefect.yaml so far

it will pull the whole repo from the

source

you pass