https://prefect.io logo
Title
m

Miguel

06/23/2020, 1:44 PM
Hi everyone I'm migrating an airflow dag to prefect and prefect server and trying to figure out how to backfill this flow. I've created a function that uses the
client.create_flow_run
to generate all the necessary runs with the correct context but, at least using the local agent, this creates way too many processes (one for each run, I guess). Is this this right approach or is there a way to limit the concurrency / number of processes spawned?
z

Zachary Hughes

06/23/2020, 2:00 PM
Hi @Miguel! We have flow concurrency limits in the works for both Cloud and Server, so watch this space on that front! In the meantime, my recommendation would be to use an agent that can scale more gracefully with your use or to use the
scheduled_start_time
parameter to help spread out the number of runs your local agent would be picking up at any given time.
m

Miguel

06/23/2020, 2:04 PM
Ok, thanks! Guess I'll override the agent query_flow_runs to limit the results based on the number of submitted / in progress runs
z

Zachary Hughes

06/23/2020, 2:05 PM
That definitely works too!
👍 1
n

Newskooler

09/24/2020, 8:58 AM
Hi @Miguel I too am migrating from Airflow to Prefect and need to run a Flow (DAG) on a daily schedule, but can’t figure out how to backfill (run the idempotent task for previous dates where each on takes a
datetime
as an input). It seems like you have figured out how to do this in Prefect. Can you please share how to achieve this? : )