@Marvin is it possible to have multiple instances of the prefect server running with a shared db instance?

To answer your question about running multiple instances of the Prefect server with a shared database instance, I need to determine the specific Prefect version you are using. This capability might differ between Prefect 2.x and 3.x. Could you please confirm the version you're using, or if you don't know, would you like me to assume a specific version to proceed with the search?

@Marvin I am running with 3.x. What is the best way to scale a self hosted prefect server?

When it comes to scaling a self-hosted Prefect server, especially in Prefect 3.x, there are a few things you can consider: 1. Running Multiple Instances: Prefect does allow you to run multiple server instances, which can be connected to a single shared database. This setup can help distribute the load and ensure high availability. 2. Proxy and Load Balancers: If you're planning to scale horizontally (i.e., running multiple instances of Prefect server), you might want to set up a proxy or load balancer to intelligently route requests to different instances. Prefect supports proxy configuration via environment variables like

HTTPS_PROXY

. 3. Concurrency Control: For managing tasks or processes that require limiting resource usage, you can use Prefect's concurrency controls. Prefect allows you to specify concurrency limits globally or on a per-task basis to prevent exhausting system resources. 4. Configuration Adjustments: Adapt the configuration settings, such as

PREFECT_API_URL

, to point to your server instances as required by your infrastructure setup. For detailed instructions, you can look into the Prefect 3.x documentation on self-hosting. This will provide more specific guidance tailored to your infrastructure setup. If you run into any issues or need more help with a specific aspect, please feel free to ask!

1. Running Multiple Instances: Prefect does allow you to run multiple server instances, which can be connected to a single shared database. This setup can help distribute the load and ensure high availability.

this is not correct what specific problems are you having when scaling? do you have any jobs to clean up old data?

Our prefect server keeps hitting rate limit issues and returning 429s causing flow runs to fail. I am currently testing with multiple flow runs that have 50 sub flows each. We don't have any jobs to clean up old data.

so oss shouldnt have any sort of rate limits

Yeah that is what we are currently looking into and seeing if any of our underlying infra is throttling us. The DB instance we have up seems fine. You mentioned that we shouldn't have multiple instances of the server running so we have 1 cloud run service running with concurrency set to 1000 but see concurrent requests get capped at 200 with 100 or so flows running.

just to make sure I understand, you're saying you're running the prefect server as a cloud run service, and for some reason you're seeing that you're only able to run 200 or so flows at a time? • what database are you using? it should be postgres not sqlite for scale • what infra are you using for flow runs?

For web sockets we see a number of 101 status requests to

<https://prefect>.../api/events/in.

and saw about 678 events for a single job submitted (that had 32 sub flows)

thats reasonable-ish for a given flow or task its like • SCHEDULED -> PENDING • PENDING -> RUNNING • RUNNING -> COMPLETED at least

We are using GCP CloudSQL postgres15 with 2 vCPUs and 16GB of memory and 10GB SSD. The DB stays below 10% CPU usage with at most 15 connections from prefect. We are using cloud run v2 to run flows. We have a GCR v2 work pool with concurrency set to 300 and a single cloud run v2 worker.

hmm yeah I guess I wouldnt say its impossible in principle, but there's some in-memory stuff that you might have to tinker with, for example this would be a great discussion to open if you don't mind, I think your original question sums it up well > is it possible to have multiple instances of the prefect server running with a shared db instance?

Are there any resources for what context or in-memory stuff prefect sets up and how to possibly configure that?