Hi @Howard Cornwell,
Looks like you’re in a situation where scale is important. Prefect Core can definitely handle that many mapped task runs. As you’ve identified, your orchestration service is the bottleneck. You have a couple of options:
1) You can try to horizontally scale Prefect Server.
This may be difficult. You’ll need additional Apollo/GraphQL instances and you’ll probably want to connect to a different Postgres instance (possible managed, something like CloudSQL) . Doing so will require changing the Docker Compose file. If you’re comfortable with all of that, give it a shot and let us know how it goes!
2) You can migrate to Prefect Cloud
Prefect Cloud has mechanisms that will help a workflow of that size scale nicely (caching, batch writing logs/states, etc). Thanks to the hybrid model, you data will never leave your infrastructure. You can try Prefect Cloud for free!