Good morning! I dont know if this is the place for questions, if It is not,sorry in advance.
We are implementing prefect in our company and I would like to ask a question.
We have deployed prefect Server on a Kubernetes cluster. We have several flows using different custom Python packages and different versions of these. The question I have is it best practice to deploy one agent per flow with a specific environment or have a single agent with all the packages and consume it from all the flows? On the other hand, is there any defined process for the CI/CD of the flows?
06/15/2022, 11:40 AM
You are in the right place! This channel is great for all questions.
You can have many agents, but having one agent per flow would be too much. You can have only one agent and separate dependencies using Docker or Kubernetes
Are you asking this for Prefect 2.0 or Prefect 1.0?
06/15/2022, 12:27 PM
Thank you very much for the quick response!
If I understand correctly the agent would be in charge of orchestrate the flows and and I could configure the environment for each of the flows and will be executed as a Kuberntes Job. Knowing this, will a Kuberntes Job be executed for each of the runs of a flow or will all the runs executed in the same pod?
Sorry for asking so many questions, we are just starting with this tool.
By the moment we are implementing prefect 1.0
06/15/2022, 12:38 PM
the agent is a lightweight process that polls for scheduled runs and then deploys it to the execution layer - your agent could run on your laptop, while your execution layer could live somewhere else e.g. a Kubernetes cluster on AWS EKS