I'm looking at a scenario where a flow could be triggered a number of times simultaneously (via the api) - I think the extreme case is around 300x within a few seconds. I don't mind the tasks queuing up, but is there a point at which something will stop working from the Prefect side of things? I need to make sure the flows will indeed run - sooner or later.
03/31/2022, 11:30 AM
Are you asking for Prefect 1.0 or 2.0? With 2.0 no issues for sure.
But even in 1.0, it should work as long as your execution layer can handle that. The UI may take some time to display all those flow runs but on the backend side, there should be no issues even in 1.0.
03/31/2022, 11:35 AM
Thanks! I guess I'm asking for both 1 and 2. The execution layer won't be able to scale (not fast enough anyways), so I guess the scenario is really about having 300+ jobs queued up safely while the agent trods along. And preferably in a first-in-first-out fashion.
I don't need a responsive UI for these edge cases, so that's good.
03/31/2022, 11:40 AM
If that's the case then probably the best option would be setting flow-level (or in 2.0 work-queue level) concurrency limits so that your scheduled runs would be queued up and then gradually picked up by the agent