https://prefect.io logo
Join Slack
Powered by
Hi everyone, I’ve just started using Prefect at my...
# prefect-community
r
Hi everyone, I’ve just started using Prefect at my company. First of all, thank you for a great product. I am trying to deploy Prefect on GCP and find the documentation very good except for GCP deployment, so I’m struggling a bit. I understand that the VertexAgent is a quite new addition. I have so far successfully registered a Vertex agent on Prefect Cloud but I don’t quite manage to deploy the agent to Vertex. Is the intention to start a Custom Vertex job in addition to 
prefect agent vertex start
? Or would you recommend deploying Prefect on in another way on GCP instead?
discourse 1
a
We appreciate your positive feedback! 🙏 Many users deploy Kubernetes Agent with GKE - e.g. here is a great tutorial by Ben Welsh Regarding the Vertex agent, check this tutorial - but you're right that running this command is everything you need (you can run it from any terminal authenticated with GCP service account):
Copy code
prefect agent vertex start --label yourlabel
more details: • https://docs.prefect.io/orchestration/agents/vertex.html#requirements • https://docs.prefect.io/orchestration/flow_config/run_configs.html#vertexrun
r
Thank you for your generous reply! 😄 I’ve re-read those documents and I finally think I grasp the overlaying architecture. Just to confirm, so the 
prefect agent vertex start
does not start an agent on Vertex but rather starts an agent where the command is run from? In that case, a small GCE or Cloud run should be enough for hosting the actual VertexAgent that will then spin up Vertex machines for the flow runs?
a
#1 Exactly! #2 Yes!
you can think of an agent as a lightweight process polling for flow runs, but in contrast to many competing solutions on the market, it's not responsible for executing the runs directly, but rather creating a separate infrastructure for the flow run (a separate Vertex job, subprocess, Docker container, Kubernetes job, ECS task)
r
Awesome, thank you for the clarification! 😄
We are now up and running with the deployment! Thanks again for the help 🙂 I thought I’d try to give something back from my learnings, coming in as a fresh user of Prefect. 1. Really love the documentation on specific concepts 2. Lacked some clearer explanations of the overarching architecture. I found this

youtube clip▾

the most helpful in explaining how the hybrid solution works. I think a deepdive documentation on just the arcitechture would help new users linking together all the different components like; Agents, Storage, Flow Runs, Executors. Something like the picture of architecture overview but in more detail 3. Great community!!
a
Thanks a lot, and great to hear you got it all working! 🙌
r
Hi again, ran into another problem with storage. I got the classic 
ModuleNotFoundError("No module named 'src'
, as mentioned in several previous slack threads and on discourse. I understand this is because storage pulls only the flow file. The ideal case for us would be to only redeploy the Docker image when updating dependencies and pull the src code from Github, much like they are doing in the tutorial that you linked earlier; only rebuild the docker image when the dependencies change, and not the src code. I understood that we could package our src code and install into the image. But is there a way around having to rebuild the docker image and rather mount the src code when deploy flow runs?
a
certainly not on Vertex since it's serverless (stateless) - mounting volumes is stateful
btw for the future - we sort of closed the issue and you started asking other questions in the same thread. It would be great if you could start a separate thread for a different question
r
Yea makes sense, will try your proposal of packaging the source code!
Yes sorry, will keep that in mind for the future 🙂 ! Thanks
a
awesome, thanks a lot! 🙌
82 Views
Open in Slack
PreviousNext
Powered by