https://prefect.io logo
e

Ethan Veres

09/05/2023, 6:57 PM
Hi all - I’m trying to migrate to Workers and using this as a chance to re-architect our flow and could use some advice. Background - We have a monorepo (hosted on GitHub) that hosts many different dbt projects (one per tenant). - Each project has a different set of parameters but ultimately runs the same flow code, just with a different working directory. - The flow code is defined in the monorepo and is built into a Docker image that is pushed to ECR. - Each tenant has their own Prefect deployment and uses the same Docker image. - Prefect pulls the tenant’s code from an S3 bucket - storage block. - Runs are using ECS Fargate. Goal - Use GitHub Repo as the
pull
source for each deployment, but with a different working directory for each tenant. - Retain the concept of a different deployment for each tenant. What would be the best way to achieve this? Split out a
prefect.yaml
for each tenant? Something else that’s more DRY? Any advice would be greatly appreciated! Thanks!
t

Taylor Curran

09/05/2023, 7:35 PM
Hi Ethan, here is a guide for upgrading from Agents to Workers 🙂 https://docs.prefect.io/latest/guides/upgrade-guide-agents-to-workers/
You will want to have one prefect.yaml per repo. Here is a great example of a prefect.yaml file that builds an image and passes it to the work pool as a
job_variable
https://github.com/EmilRex/kubernetes-worker-demo/blob/main/prefect.yaml
e

Ethan Veres

09/05/2023, 7:45 PM
Thanks @Taylor Curran! When you say per repo do you mean per tenant? Or just one top-level one that somehow deploys for each tenant
t

Taylor Curran

09/05/2023, 7:47 PM
Currently 1 prefect.yaml file per github repo is support that would deploy for each tennant.
Different deployments can have different pull steps.
You can create definitions like shown in line 36 of this example. Maybe a pull definition for each tenant would be a good idea, this way you can reuse them across multiple deployments
e

Ethan Veres

09/05/2023, 7:58 PM
Thanks for all this! Is there any way to define all this in code instead of yaml? I fear just appending all deployments to a single file may get out of hand. We currently have about 50 deployments
@Taylor Curran any thoughts?
t

Taylor Curran

09/06/2023, 1:45 PM
We don’t support worker based deployments in python — I believe its on the roadmap to do this however
2 Views