I have a flow that I want to use to backfill data for about 9 years which is going to equal around 3000 jobs. I am curious if Prefect Cloud will have any issues queuing up so many jobs and then just running them sequentially.

Hey Jeff! Absolutely no issue with that scale; however, I do want to call out one thing — let’s say you use the API to create 3,000 runs, all with the same start time. Unless you layer on a concurrency limit for the flow’s label, your agent may attempt to submit them all at once. From Cloud’s perspective that’s fine, but your agent’s infrastructure might not like it so much.

Very good point. We were going to enable a concurrency limit on that flow for sure. We don’t have 3000 servers excited to work on this.