Hey you all,
I’m new to Prefect and I’ve managed set up a dev & deployment process for my team to start working with Prefect development in production, however I’m still trying to find a good solution to manage the Agents and automate that part of the deployment as well.
The current deployment workflow (done in Github Actions) looks roughly like this:
• Build & Push new Docker image (dev/prod) whenever project requirements in the related branch are changed
• Build & register flows to Prefect Cloud (dev/prod project) whenever flow files (or default build configs) in the related branch are changed
We currently have two Docker Agents running in an AWS EC2 instance. Each agent has its own environment variables and currently, if new env variables need to be added, this is my process:
• SSH to EC2
• Start new agents with new environment variables e.g.
Nohup prefect agent docker start --label dev --env VAR1=<variable1> --env VAR2=<variable2> &
• Get the PIDs for the old agents with old environment variables
ps -ef | grep prefect
• Kill the old agents
kill <PID>
As this is not a very smooth approach, I’d like to hear how others have approached automating the Agent part when setting up CI processes. Any tips?