Is there a good example of having classes defined in a separate file and using that class in a task (on Kubernetes and Azure) - I keep getting the message module not found

@Jai Deo last week I’ve built an example you’re looking for! 🙂 Some explanation: • the flow runs on Kubernetes - thus we need

KubernetesRun

• since you have custom module dependencies, those need to be packaged into a Docker image that gets built and pushed to Azure Container Registry - README of this repo explains how to do that and Dockerfile shows how the image can be built • the custom package is shown as

flow_utilities

in the flow https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/github_kubernetes_run_custom_azure_image.py

Hey @Jai Deo, Anna is right here. All flow storages except for Docker and Module only serialize the flow code, not the dependent modules so you would need to package them in Docker to run on Kubernetes

Hi Kevin, I don't understand - that sounds like I can't store the flow in azure ( if I have separate packages in separate files ?)

Yeah for Azure Blob storage you can’t because of how

cloudpickle

works where it doesn’t deepcopy the modules. With

cloudpickle 2.0

recently released, there is support for deep copying of modules which might make this possible.

In Anna's example she uses github as storage and specifies the path as an option - azure does'nt have this option does it ?

Azure has a path but for both Azure and Github, only the flow file is loaded.

setup.py

in Docker will work. We have a more generic

Git

storage that clones the whole repo and we talked about installing the repo as a module, but it felt like reinventing Python packaging ourselves. If you have the

setup.py

, you can install your dependencies in a Docker container to be used for the flow.

@Jai Deo • for Python dependencies, you need a Docker image. • For flow code, you can use e.g. Azure blob storage or Github. • Alternatively, you can bake your flow code into the image and use it with Docker or Local storage

I will have to ponder on the options.... Thanks