https://prefect.io logo
Title
m

Mark McDonald

10/24/2019, 8:15 PM
Hi - I'm looking for some feedback on how we're deploying our flows. I'm trying to dynamically deploy our flows using Docker and a custom deployment script. Within my project, I have a module that contains my flows (each flow in a separate file). Each of these flows has various dependencies that get imported (pandas, scipy, numpy, etc.). We pin our dependencies in a requirements.txt file. My deployment script does the following in this order: 1. Grabs all of my flows' dependencies from the requirements.txt file and adds it to a list called
dependencies
2. Installs these dependencies on my docker image 3. Builds prefect docker storage object: storage = Docker(registry_url=os.getenv('PREFECT_ECR_REGISTRY'), image_name=os.getenv('PREFECT_ECR_IMAGE_NAME'), image_tag=package_version, python_dependencies=dependencies) 4. script imports each of my flow files and locates the flow objects within them. These flows get added to the storage object 5. calls storage.build 6. iterates through the flows and calls flow.deploy() on each flow object with build set to False As it stands the deployment takes ~5 minutes. Any areas where I might be able to improve this?
✔️ 2
c

Chris White

10/24/2019, 8:51 PM
Hi Mark, you could pre-build a base image that has all the python dependencies pre-installed, but that assumes slow-moving dependency changes
m

Mark McDonald

10/24/2019, 9:23 PM
got it - yea I'll consider that. thanks, Chris