https://prefect.io logo
Title
h

Hawkar Mahmod

08/05/2020, 11:55 AM
Hey folks. I've got a Prefect flow set up. For execution I am using the Fargate Agent, itself hosted on an ECS cluster. In the flow I've configured the Docker Storage to use a custom ECR repository to host the image for the flow and I then register the flow in the same module (.py file). I am however ensure how to coordinate the deployment of the agent and new flows. How are people registering flows in production and deploying their agent? Is it all part of one build process, or are they separate, and if so how do you manage it, what tools are you using for instance? I am considering using AWS CodeBuild since we already use it on other projects?
j

josh

08/05/2020, 12:39 PM
Hi @Hawkar Mahmod I encourage others in the community to mention how they deploy and manage their agent/flows but I can offer some suggestions on how I have seen it done. Spinning up the agent either manually or through some sort of automated process (like Terraform) on an EC2 instance is something I commonly see. Not too familiar with CodeBuild but I have seen that name thrown around a bit too. As for the flow registration I have seen teams do it in a couple ways like engineers being responsible for registering new versions or (in an automated way) registering the flows in some sort of CI/CD process.
h

Hawkar Mahmod

08/05/2020, 12:42 PM
@josh thanks for that. That's somewhat reassuring since it means there isn't a single preferred method and its quite flexible. Welcome contributions from others.
t

Thomas Hoeck

08/05/2020, 12:48 PM
@Hawkar Mahmod we are currently manually deploying a k8 agent into cluster. We use Azure DevOps Pipeline to build all flows into one image when merged into master and register these through the pipeline (including pushing the image to Azure Container Reg). This limits the number of images stored in ACR.
m

Mark McDonald

08/05/2020, 12:48 PM
We keep flow and agent deployment separate. We're using AWS Codebuild. We have flows stored across a few Github repos. Merging to master deploys the flows to a UAT project in Prefect cloud. Releases deploy the flows to a Prod project in cloud. We have a separate repo that stores all of our terraform and deploys the agent as a Fargate/ECS task.
m

Michael Ludwig

08/06/2020, 6:11 AM
We are doing it very similar as @Mark McDonald. We use GitLab CI and CloudFormation though
h

Hawkar Mahmod

08/18/2020, 3:23 PM
Hi thanks all for your input. Since we only have one flow currently, and one person working on it all we've decided to deploy the flow and agent together. We'll be using AWS Codebuild as well, with flows being registered upon each deployment of the repo.