Hi everyone I'm migrating an airflow dag to prefect and prefect server and trying to figure out how to backfill this flow. I've created a function that uses the

client.create_flow_run

to generate all the necessary runs with the correct context but, at least using the local agent, this creates way too many processes (one for each run, I guess). Is this this right approach or is there a way to limit the concurrency / number of processes spawned?

Hi @Miguel! We have flow concurrency limits in the works for both Cloud and Server, so watch this space on that front! In the meantime, my recommendation would be to use an agent that can scale more gracefully with your use or to use the

scheduled_start_time

parameter to help spread out the number of runs your local agent would be picking up at any given time.

Ok, thanks! Guess I'll override the agent query_flow_runs to limit the results based on the number of submitted / in progress runs

That definitely works too!

Hi @Miguel I too am migrating from Airflow to Prefect and need to run a Flow (DAG) on a daily schedule, but can’t figure out how to backfill (run the idempotent task for previous dates where each on takes a

datetime

as an input). It seems like you have figured out how to do this in Prefect. Can you please share how to achieve this? : )