So, at a new company and helping them set up their...
# ask-communitym
matta
04/23/2021, 1:53 AMSo, at a new company and helping them set up their data infra! What would be the smoothest way to get up and running with Prefect Cloud on GCP and with a pretty small team? Running the flows on GKE?
matta
04/23/2021, 1:55 AMFor the moment, mostly just running stuff with Cloud Functions and Cloud Run. For the moment, would probably still be calling those from Flows I think (just from Prefect, with a nice centralized scheduler).
n
nicholas
04/23/2021, 1:57 AM
Hi @matta - welcome back! GKE seems like a good place to ramp up and it seems like there are a lot of people in the community using it (and therefore can probably help out if you run into issues!)
k
Kevin Kho
04/23/2021, 1:57 AM
Hi @matta! Congrats on the new job!
m
matta
04/23/2021, 1:58 AMThanks!
matta
04/23/2021, 2:04 AMI guess the main bottleneck for me right now is where exactly to run the Agents and register the Flows and stuff? I guess just a small Compute Engine instance?
n
nicholas
04/23/2021, 2:06 AM
You could run the agent on GKE as well, which would be especially good if you're including it in a CI process
nicholas
04/23/2021, 2:07 AM
but using a dedicated instance is fine as well, just potentially more annoying to maintain
m
matta
04/23/2021, 2:09 AMRight on! Hrm, for Registering flows, maybe a Cloud Function that's kicked off by a GitHub Action?
matta
04/23/2021, 2:11 AM(or I guess maybe GKE again?)
n
nicholas
04/23/2021, 2:13 AM
Hm good question, Cloud function might be better for that
m
matta
04/23/2021, 2:16 AMCloud Run seems a little simpler than full-blown GKE for the moment actually - would I be able to run the Agents and stuff from there? I guess from the Docker Agent?
n
nicholas
04/23/2021, 2:26 AM
That's something we want to make viable but isn't just yet - if you're going scaling back a little I'd say that's when using a smallish compute instance might be best, so long as you're not running the flows locally (you'll hit resource limits really quickly)
m
matta
04/23/2021, 4:31 AMRight on! Thanks!
2 Views