https://prefect.io logo
Title
p

Pedro Machado

07/10/2020, 5:48 PM
Hi everyone. Is my understanding correct that when dealing with multiple python files (ex: one file has a flow that imports objects from another python file) the choices are: 1. use docker storage passing the
files
dictionary 2. use a custom docker image with everything needed to run the flow 3. packaging the additional python files and pip installing them on the machine(s) that will execute the flow Am I missing other approaches? I don't think it's possible to package the flow + extra files when using the other types of storage. Correct? Are there plans to support this?
n

nicholas

07/10/2020, 6:26 PM
Hi @Pedro Machado - you're correct, those are the ways to deal with multiple Python files; I don't think there are plans to support a Prefect package manager at the moment but I'd encourage you to open a ticket with your expectations for a feature like that 🙂
p

Pedro Machado

07/10/2020, 6:31 PM
I was wondering if there is a native way to support for example zip files. I believe airflow can read dags from a zip, but it's something I haven't used. This is not a pain point for me now as I am getting familiar with Prefect but I did run into it when I tried to move from my local environment to running an agent on AWS. My initial attempt was to use S3 storage but then I realized that it only serialized the flow and not the other file. I do have one suggestion on the Docker storage documentation. There is little info about how you should use
files
I had to look at the code to understand what values to pass to the dict.
n

nicholas

07/10/2020, 6:36 PM
Definitely @Pedro Machado! You can open a GitHub ticket directly from the docs by highlighting the text you're finding confusing and clicking the GitHub icon; feel free to open a PR for the docs as well!
g

Greg Desmarais

07/10/2020, 9:20 PM
@Pedro Machado - are you storing your dockers in ECR?
p

Pedro Machado

07/11/2020, 2:22 AM
Yes, that's what I did for the flow I was testing.
Nicholas, I'll submit a PR for the documentation as soon as I get a chance. Thanks
n

nicholas

07/11/2020, 2:34 AM
No worries @Pedro Machado, thank you! If you're feeling it's a bit tougher than you thought feel free to open an issue and someone on the Prefect team will tackle it, we really appreciate your help either way!