Hi, I am in the process of evaluating Prefect Cloud for my pipeline. I have a setup where there is a flow definition with parameters, and I have about 1000 parameter sets from database (and a lot more when the pipeline is live). What I can do is to deploy the flows with parameters for each parameter set via API. I want to know how this scenario can be handled: If I update the code of my flow, such as using a different algorithm, or add another step, how can 1000+ deployments be updated?

Are you running these all as separate flows?

I plan to use the Prefect scheduler to run these jobs

What storage are you using? I am thinking you might not even need updating depending on your DeploymentSpec if you just point to the file

I think S3

Ah for S3 yeah I think so it will make a copy but maybe you only need to update the file once?

I don’t understand the “only need to update the file once” part 🙂

It would be more helpful for you to keep the discussion so someone else can pick it up from our side because tomorrow is my last day here 😅

Ah ok 😄

But I mean, let’s say you re-create 1000 deployments. Does it just fial?

So suppose I have this flow

@flow
def forecast(customer: str, dimensions: List[str], metrics: List[str]) -> pd.DataFrame:
    data = load_data_task()
    results = [forecast_task(df) for df in data]
    return results

Now, I have a web service that when I receive a new forecast setup, then I would invoke

deployment = Deployment(flow=forecast, params=params)
await deployment.create()

The deployment should be saved to S3. I have 1000+ deployments like this, and lets say I make changes to the

forecast

flow, or to the tasks, then how can this update be propagated? Considering that I am using Prefect Cloud.