Hey there, first time here.
Firstly, fantastic product you have built, seems to have massive potential, coming from someone who used to work with Airflow. The documentation has been super helpful and I would like to see more video demos in the future.
My question is, what would you recommend would be the best way to run Agents in a production environment, say EC2, when using Prefect's Cloud feature?
• should the Agent run continuously in the background of an EC2 instance?
• should the Agent be embedded in the Python Flow script and run as a container
• should we have several Agents running and associated to flows based on
Basically, some best practices around Agents in a production setting.
04/06/2020, 2:54 PM
We've been using Prefect Cloud for awhile, and it's really flexible to your use cases.
For us, this involved having a handful of Agents running in different environments. Some of our Flows require access ot on-prem resources, so the on-prem K8 Agent picks up those Flows. Other flows simply need a ton of compute resources, so those are triggered in GKE with an Agent listening in there. Then other non-critical Flows are ran in a laboratory K8.
Spawning all the Agents on K8 was very simple to get started, and it's been pretty stable for us.