hi gang! what's the best practice way of registering flows that aren't contained to just one

.py

file? we've split out commonly shared functionality between flows into an e.g.

utils.py

file which is referenced in the flow. given all the storage documentation this design doesn't seem to fit into the intended use of storage.

Hey @Constantino Schillebeeckx, Docker storage is the recommended approach where you would couple all of your dependencies into in the image so that they become available at runtime. This also might work with Module storage where if you package everything up as a Python module and install it on the agent, you could run those flows. I have an example of module storage here . The downside though is that it’s a bit easy to become out of sync between the module on the agent and development (unless the agent is Local). If

utils.py

does not change often, you can have it as a Python module inside your Docker container and then store your flow somewhere else like S3. S3 + DockerRun will pull the flow from S3 and run it on top of the specified container. This way, you won’t have to keep rebuilding containers.

So if I understand correctly, I might do the following: Create a custom docker container with all my extra requirements as well as my custom code (e.g.

utils.py

, reference this container in my ECSRun, and then use S3 for just the flow's

workflow.py

which is able to e.g.

import utils

?

https://github.com/PrefectHQ/prefect/discussions/4042 may be useful as well 🙂

yessss my next question was around CD 🙂 thanks for the resource, I'll post back here if I've got any other questions

Yes that’s what I meant!

side question: as I'm developing on ECR and ECS, and I'm pushing changes, testing things out, it feels like some "things" are getting cached. e.g. I'm getting a failure like

ModuleNotFoundError("No module named 'pipelines.custom_docker'")

when I've since removed all references to that line of code, pushed new containers and re-registered the flow

Is your image tagged to latest and did you register with the latest image? What storage are you using? DockerStorage?

yes and GitHub

As in ECSRun and Github storage?

ooo I have to push the change 🙂

Yeahhh. Probably that 👍

As in ECSRun and Github storage?

correct

How did you install in the module?

pip

or

conda

?

pip

Do you have an environment that might not be used? Or you just installed everything with pip?

everything is installed with pip; no virtual environment is used in the Docker container

Could you show me the Dockerfile?

Can you show me the ECS RunConfig?

Everything looks alright. I think what might be happening is they might be installed in different locations inside the container like this . Maybe you can try

pip install .

instead of the

python setup.py install

to make sure? Is

./flows

everything in your package? I don’t believe there should be caching if you are running as a task with Fargate/ECS. If you did on EC2 then there might be, but you could tag to be explicitly sure? How did you test the container to see if you could import? You downloaded your image and then used

exec

?

Let me try

pip install .

. Yep,

./flows

is all my code; that's what setup.py references

You might also be able to test this locally with a Docker Run and Docker agent if that makes testing easier

I've been able to reproduce it locally - looks like you're right about my install being wonky.

Can I see your setup.py? (I’m just asking for everything at this point lol) I have a suspicion. I ran into a similar thing two days ago

It looks right to me. Maybe try

packages=find_packages()

instead of

['flows']

, but this would require

___init__.py_

files i think in those subdiractories

FYI,

__init__.py

only in the highest level dir

I see, maybe the init in the sub dirs will help

find_packages()

get them? Are you python versions the same? At least we know the culprit

you rock! it was a combination of using

find_packages()

and then having an

__init__.py

in every subdir

Nice! Yeah I had to do it but don’t know why 🤷