I'm using process workers inside of Docker containers. I have a flow that runs on an interval and checks for events to process (which has already gone a filtering process). What is the preferred method of distributing this work to the workers? I'm currently using

client.create_flow_run_from_deployment

to create a new flow run for each of these but the overhead of starting all of these flow runs appears to be too much. Is there a faster way that I'm not aware of, perhaps some sort of create-batch?

What sort of flow run volume are you looking to start at once, and what's the issue you're experiencing?

I'm looking to start around 2 000 flows from another flow. From the graph in the UI it seemed like the submitting the flows to their deployment was taking ~1.5 seconds each.

Yeah, there isn't anything like that right now that I'm aware of. If you want event-driven flow run triggers, automations in Prefect Cloud are probably your best bet. You could emit events from your source system or hit a webhook, either of which would trigger an automation that starts your downstream flow run.

But those would both have the same issue, no? Seems like I would have to issue an HTTP request for each event in that case as well.

Right, but you could emit events to trigger the automation as they happen, rather than batching them over some interval of time. The emitting part wouldn't have to be inside an existing flow, but the place the events are actually happening.

Yeah, that's a fair point. Do you think there would be any interest in a PR for batch-endpoints? If I wanted to take a stab at implementing it.

I think it's an interesting idea, but it's hard for me to say how desired this is. Maybe open an issue or feature request to get the conversation going.