https://prefect.io logo
Title
t

Toby Drane

04/18/2023, 4:46 PM
Hi all, I have a AWS ECR Docker Registry and don't know if it possible to use this as a storage for Prefect deployments. If I write different flows and deployments is it possible to build a Docker container for each of the flows and use ECR as a way for deployment to store the logic?
r

Ryan Peden

04/18/2023, 4:57 PM
That should work. You might want to try the new projects feature for this. If you check the projects docs you'll see a section about automatically building and pushing Docker images with the Docker recipe: https://docs.prefect.io/latest/concepts/projects/. Going a little further, this tutorial might be exactly what you need: https://docs.prefect.io/latest/tutorials/projects/#dockerized-deployment I believe that will work for you if you've authenticated Docker with your ECR registry on the machine where you build the deployment. On the runtime side, all the Prefect flow run infrastructure (both agents and workers) that supports Docker images should also support private registries, ECR included.
t

Toby Drane

04/18/2023, 5:32 PM
Is it possible without Prefect projects? I am hosting all Prefect architecture myself aswell on AWS, i.e. the servers and agents
🙌 1
r

Ryan Peden

04/18/2023, 5:42 PM
Sure, one approach that comes to mind is: • Build a container image for each deployment; when building the image, copy your flow files to
/opt/prefect/flows
inside the image. That's where Prefect will look for it by default when running on infrastructure that uses Docker images. â—¦ You can set a different path using the
path
attribute of your deployment. ◦ If you're curious, here's how Prefect decides where to look for your flow code in this case: https://github.com/PrefectHQ/prefect/blob/2f65d50cbeb66f91c7375b6a55c9e0d9f62bd011/src/prefect/deployments.py#L786 • Push the image to ECR • Build your Prefect deployment without storage; since you've already baked the flow code into the Docker image, you don't need external storage • When you set up your infrastructure, add a
DockerRegistry
block with credentials for your ECR registry. As mentioned above, all the flow run infrastructure types that run Docker images should have a registry field that lets you add a private registry â—¦ When you're using a private registry, you'll need to specify the full URL of the image you use, e.g.
my-registry.ecr.aws/abcdefg/my-flow:latest
🙌 1