Rasmus Lindqvist

    Rasmus Lindqvist

    5 months ago
    Hi everyone, I’ve just started using Prefect at my company. First of all, thank you for a great product. I am trying to deploy Prefect on GCP and find the documentation very good except for GCP deployment, so I’m struggling a bit. I understand that the VertexAgent is a quite new addition. I have so far successfully registered a Vertex agent on Prefect Cloud but I don’t quite manage to deploy the agent to Vertex. Is the intention to start a Custom Vertex job in addition to
    prefect agent vertex start
    ? Or would you recommend deploying Prefect on in another way on GCP instead?
    Anna Geller

    Anna Geller

    5 months ago
    We appreciate your positive feedback! :thank-you: Many users deploy Kubernetes Agent with GKE - e.g. here is a great tutorial by Ben Welsh Regarding the Vertex agent, check this tutorial - but you're right that running this command is everything you need (you can run it from any terminal authenticated with GCP service account):
    prefect agent vertex start --label yourlabel
    more details: • https://docs.prefect.io/orchestration/agents/vertex.html#requirements https://docs.prefect.io/orchestration/flow_config/run_configs.html#vertexrun
    Rasmus Lindqvist

    Rasmus Lindqvist

    5 months ago
    Thank you for your generous reply! 😄 I’ve re-read those documents and I finally think I grasp the overlaying architecture. Just to confirm, so the
    prefect agent vertex start
    does not start an agent on Vertex but rather starts an agent where the command is run from? In that case, a small GCE or Cloud run should be enough for hosting the actual VertexAgent that will then spin up Vertex machines for the flow runs?
    Anna Geller

    Anna Geller

    5 months ago
    #1 Exactly! #2 Yes!
    you can think of an agent as a lightweight process polling for flow runs, but in contrast to many competing solutions on the market, it's not responsible for executing the runs directly, but rather creating a separate infrastructure for the flow run (a separate Vertex job, subprocess, Docker container, Kubernetes job, ECS task)
    Rasmus Lindqvist

    Rasmus Lindqvist

    5 months ago
    Awesome, thank you for the clarification! 😄
    We are now up and running with the deployment! Thanks again for the help 🙂 I thought I’d try to give something back from my learnings, coming in as a fresh user of Prefect. 1. Really love the documentation on specific concepts 2. Lacked some clearer explanations of the overarching architecture. I found this

    youtube clip

    the most helpful in explaining how the hybrid solution works. I think a deepdive documentation on just the arcitechture would help new users linking together all the different components like; Agents, Storage, Flow Runs, Executors. Something like the picture of architecture overview but in more detail 3. Great community!!
    Anna Geller

    Anna Geller

    5 months ago
    Thanks a lot, and great to hear you got it all working! 🙌
    Rasmus Lindqvist

    Rasmus Lindqvist

    5 months ago
    Hi again, ran into another problem with storage. I got the classic
    ModuleNotFoundError("No module named 'src'
    , as mentioned in several previous slack threads and on discourse. I understand this is because storage pulls only the flow file. The ideal case for us would be to only redeploy the Docker image when updating dependencies and pull the src code from Github, much like they are doing in the tutorial that you linked earlier; only rebuild the docker image when the dependencies change, and not the src code. I understood that we could package our src code and install into the image. But is there a way around having to rebuild the docker image and rather mount the src code when deploy flow runs?
    Anna Geller

    Anna Geller

    5 months ago
    certainly not on Vertex since it's serverless (stateless) - mounting volumes is stateful
    btw for the future - we sort of closed the issue and you started asking other questions in the same thread. It would be great if you could start a separate thread for a different question
    Rasmus Lindqvist

    Rasmus Lindqvist

    5 months ago
    Yea makes sense, will try your proposal of packaging the source code!
    Yes sorry, will keep that in mind for the future 🙂 ! Thanks
    Anna Geller

    Anna Geller

    5 months ago
    awesome, thanks a lot! 🙌