Alvaro Durán Tovar
04/29/2022, 9:19 AMfor flow in flows: flow.storage = docker
) and register the flows. This allows for easily reuse the same image, dependencies, etc.
Now I've started a new project integrating feast and I want to have some flows to materialize the views. That means I guess having 1 flow per thing I want to materialize, then following the same pattern as above but on a different project. Starts to be a bit of too much repetition (although it works well tbh).
Is other people doing something similar? How do you deal with distributing the code for the flows? I can only find using docker images because the flows often have dependencies on other python modules.prefect-flows
project
Now I have and I want to tell prefect to use the latest version of this project when executing the flows. Maybe I can:
1. Have the flows in prefect-flows project.
2. On feast project build the docker image and call prefect api to tell there is a new image.
Prefect will just use the latest configuration one way or another 🤷
In this possible @Anna Geller? Modify the storage "on demand" through the api?Anna Geller
04/29/2022, 11:25 AMHow do you deal with distributing the code for the flows?One idea worth exploring is using flow storage purely for flow code (e.g. using S3 or GitHub) and then packaging code dependencies via a separate mechanism. The most convenient approach is building a custom package since for local development, you can even install that in editable mode if you want, and use the same logic to python code dependencies for production by installing the same package within your Docker image
In this possible @Anna Geller? Modify the storage "on demand" through the api?Technically, it may be possible since the reference to the storage is stored in the backend as flow metadata (as opposed to e.g. executor which is read from storage at runtime). But we don't expose any hooks to do that and the solution might get convoluted, so it may be easier to avoid that route. But if you want to, I can check the source code and GraphQL API docs to dive deeper btw, I wouldn't focus too much on building best practices for that in 1.0 - this will be easier to accomplish in Prefect 2.0 due to the separation of concerns here - you would define such information separately from flow in a
DeploymnetSpec
. So I'd be more curious to discuss how to approach this in Prefect 2.0 if you want to chat moreAlvaro Durán Tovar
04/29/2022, 11:35 AMAnna Geller
04/29/2022, 11:48 AMAlvaro Durán Tovar
04/29/2022, 11:51 AM