Hi, I’m looking for some guidance on how to use the same flows across multiple projects. For context: we are doing ETL for customers and some flows are identical. We’re trying ECS Agent/Prefect Cloud with a custom Docker image and S3 storage. I am currently registering the flow with the CLI for multiple customers and passing in customer-specific information as Parameters when scheduling the run. This currently breaks down in a few places: 1. Having some problem with Storage and unpicking flows. Is there project-specific info stored there? So when a flow is registered/picked for one project it can’t be unpickled by another? 2. Using secrets. We have some customer-specific API Keys that need to be stored as Secrets and these appear to only be set globally in Prefect Cloud (or AWS SSM). There are no per-project secrets that I can pass in. One option I can think of would be to make a service that takes a Parameter and returns a secret that then gets passed to a task but maybe there’s a better way? I’m still pretty new to Prefect but couldn’t find answers to these. Hoping for some help from the community!

Your thinking is definitely right on 2. Secrets are on the tenant level so you need to do something like prefixing these API Keys and then pulling the appropriate secret. What is your error for 1? I don’t think there is project-specific information stored there. We have some people on the set-up you described

The problem on #2 is finding the name of the secret given it’s different per project. The best way to do this feels like creating another task based on a Parameter that looks up the Secret name and also get the value. Does that make sense?

It does, but I think it can be fixed by standardizing the secret names?

USER1-KEY

,

USER2-KEY

so that if you can just do param+_KEY. There is a thread here I think will be useful one sec

Thanks. This is helpful. I think the new task is going to work best for us in this case. I’m going to use AWS SSM secrets and have a naming convention that maps to a Parameter we can set in each project.

You can certainly use existing ones. Let me give you a sample script

Thanks, this looks directionally what we’re after. However, the workflow we’re looking for is to add these flows to new projects without changing the code. I’ve been relying on the CLI to register flows and that seems like it won’t be possible with this approach. Is the build = False option still not available in the CLI? Do you have other ideas for how we can add this to new projects? (BTW, we also have multiple flows in one PY file. Will also test to make sure that works.)

I feel this can still be done if you store as a script that is parameterized. When Prefect loads it, it will use the script inside the container then add parameters. Unfortunately, it doesn’t seem to be available on the CLI.

Cool. Makes sense! I’m trying to get this into a CI/CD pipeline so I created a script called register.py that gets run after the image is built anytime code is changed. It does something like this:

if __name__ == "__main__":
    flow_list = [setup_flow, semester_flow, daily_flow, hourly_flow]
    project_list = ["client_1", "client_2"]
    for flow in flow_list:
        docker_storage.add_flow(flow)
        for project_name in project_list:
            flow.register(project_name=project_name, build=False)

I’ll need to add a way to parameterize to add more projects without a build in the future. I guess the way to do that is to look at the cloud API?

I don’t have an immediate answer. I will think about this more. Ping me for sure if I don’t respond to you tom.

No worries! There’s really no urgency here. At some point we’re going to want our management tool to be able to create projects and add flows with schedules to them. We’re unblocked though and all that is a bridge for another day. Thanks for your help, Kevin!

Yep will respond a bit later

Thanks, Kevin. I’m imagining that the CI/CD process will seed defaults but that there may need to be some other systems that are doing ongoing changes to parameters. That’s why I was interesting in the Cloud API. At any rate, we’re unblocked on this for now. Thanks!

Gotcha