Hi! We have a way we are using prefect that requir...
# best-practices
z
Hi! We have a way we are using prefect that required a little bit of a hack to get working, and I'm wondering if people here can tell me if there's a better way! The issue is: we have a repo that contains some commonly used prefect flows (ill call it 'flow repo'), and we want to import those flows into several different repos, and deploy the flows from those repos (ill call them end-user repos). Our method is to define Deployment objects in the end-user repos and
.apply()
them. (we use s3 storage, and run our flows via Kubernetes jobs, in images which have the flow repo installed as a python package. This works fine except that when we build the Deployments in end-user repos, we run into the problem that the files containing the imported flows are not under the same working directory that we are building the Deployments from. The flow repo is installed as a python package and thus is under
/opt/env/.../site-packages/...
. This causes the code that sets the Deployment's entrypoint to fail with an error like
<path_to_flow_file> is not in the subpath of <cwd> OR one path is relative and the other is absolute.
. We are working around this by importing the flows and modifying their
.___module___
property to 'trick' prefect into thinking that the flows come from the end-user repo. Is there something I'm missing here that would allow us to do this in a less hacky way, or is this pattern just not something prefect supports? Thanks!