I'm looking at a scenario where a flow could be triggered a number of times simultaneously (via the api) - I think the extreme case is around 300x within a few seconds. I don't mind the tasks queuing up, but is there a point at which something will stop working from the Prefect side of things? I need to make sure the flows will indeed run - sooner or later.
a
Anna Geller
03/31/2022, 11:30 AM
Are you asking for Prefect 1.0 or 2.0? With 2.0 no issues for sure.
But even in 1.0, it should work as long as your execution layer can handle that. The UI may take some time to display all those flow runs but on the backend side, there should be no issues even in 1.0.
h
Henning Holgersen
03/31/2022, 11:35 AM
Thanks! I guess I'm asking for both 1 and 2. The execution layer won't be able to scale (not fast enough anyways), so I guess the scenario is really about having 300+ jobs queued up safely while the agent trods along. And preferably in a first-in-first-out fashion.
I don't need a responsive UI for these edge cases, so that's good.
a
Anna Geller
03/31/2022, 11:40 AM
If that's the case then probably the best option would be setting flow-level (or in 2.0 work-queue level) concurrency limits so that your scheduled runs would be queued up and then gradually picked up by the agent
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.