Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi Prefect team! How far away in the timeline for programmatic deployments with pull steps configurable? I need to configure a pull step for each individual flow/domain, as I encapsulate my flows in domain specific projects.

Hey <@U05HHPHKEA3>! Can you elaborate what you mean by programmatic deployments? You can define pull steps for each deployment defined in a `prefect.yaml` file. That may work for your use case, but if you share more information I might be able suggest a better solution for your situation.

image.png

Thanks for your reply <@U02GE39K551>! From what I understand, I will have to define a git clone pull step for each project (git repo) which contains the flows I need. If I have 30 distinct repositories, n amount of customers/tenants/environments for each repository, then potentially I am downloading 30 git repos every time a deployment runs with the way prefect.yaml currently works, even though only one Git repo is needed to run the code. Prefect.yaml will also become very large and unmaintainable.

As a work around, I've taken a page out of the GitOps+infrastructure as code philosophies and have a MAD.Prefect.Deployments project - which represents all the deployments for a Prefect server. It has soft links to all my flows, which I can configure on a per customer basis (see screenshots). It also utilizes Poetry as a dependency manager for installing required dependencies on startup of the job.

This allows me to define deployments (almost) programmatically using python, simplified DevOps by using Git based storage (to a certain branch). Ideally, I would prefer to be able to define the deployments using python with the same API that prefect.yaml uses (with pull step configuration).

Thanks for the info! It looks like you’re using a custom image for your deployments, which makes me think the `flow.deploy`/`deploy` utilities we released yesterday might work well for your use case. There’s an example of how to use `flow.deploy` in the <https://docs.prefect.io/2.13.8/tutorial/workers/|tutorial>. It doesn’t natively support post-pull actions like your `ExtendedGitRepository, but you can` build images for your deployments with your dependencies installed inside. We’re considering adding a post-pull hook to `flow.deploy,` and I’d be very interested to hear what your ideal interface for that would be since you built a version of it yourself.