Hey all, I’m in the process of migrating our prefect setup to GCP. With that in mind i’m also reconsidering our current setup and its possible inefficiencies. Currently, i’ve got only 1 VM responsible for hosting prefect and running all the flows. The computational processes are all being done in that one VM. Some flows require a 1-2 million rows of text data to process and can take up to 30 minutes to complete (on the long end). Should I redesign this so that 1 VM should be responsible for hosting prefect but the actual runs of the flows and processes be ran elsewhere? And/or should the specific heavy tasks of a flow be ran elsewhere? Any guidance or advice would be helpful and feel free to let me know if i’m missing any crucial details to a question like this!

Would def assign a separate server and worker. We host ours in the same VM but the resources are managed by Kubernetes. Maybe GKE with Google cloud run ?

Ohh i see. So with the same VM approach, you use kubernetes in order to run either flows or tasks in specific infrastructure? @Deceivious

Nah we run 3 VMs in different geographics areas for outage protection but yesyou can do that

ohh i see. Kubernetes would probably be overkill on my end but i’m not actually sure how to actually judge that, Maybe just 1 vm for the server and 1 vm for the worker. How would you typically combat running multiple computationally expensive flows around the same time on that 1 VM worker? Or ideally, would you spin off separate infra for each flow run?

yes id just a small server and maybe do the higher work load on the container instance

ahh okay.

GKE is just googles kubernetes

yeah.

depends on how many workers are connecting and how many API calls are being sent, Try small and extend as you go on

oh, hmmm… @Deceivious i actually don’t remember setting up a database for my last prefect setup. Is the database used to store metadata?

Prefect creates a SQLite database automatically when not specified. Not recommended for prod env.

gotchya- it mustve been the sqlite db then.

8gb ram is over kill imo

ahh i see. yeah, i wasn’t too sure on that front. i’ll size that down. Yup, it’s for the server.

Yes just scale stuff after

in terms of the db- we’ve got postgresql databases that we use so i’d probably just use that.

Create a new database and use existing infra yes

yup.

I think mostly concurrency and system load

ahh okay. Do processes that benefit from parallelism (using Dask or something) play a factor?

Not that way. I mean more like you don't want 2 heavy processes to start in ur infra cuz it might not be bulky enough so you might want to limit concurrency

gotchya.