https://prefect.io logo
Join Slack
Powered by
hi! I would like to understand a bit more how is p...
# best-practices
a
hi! I would like to understand a bit more how is people calling the code for the flows. So far what we are doing is having 1 repo with many flows. Then a build script that will create 1 single docker image, then assign that single docker storage to all flows (basically like 
for flow in flows: flow.storage = docker
) and register the flows. This allows for easily reuse the same image, dependencies, etc. Now I've started a new project integrating feast and I want to have some flows to materialize the views. That means I guess having 1 flow per thing I want to materialize, then following the same pattern as above but on a different project. Starts to be a bit of too much repetition (although it works well tbh). Is other people doing something similar? How do you deal with distributing the code for the flows? I can only find using docker images because the flows often have dependencies on other python modules.
➕ 1
Actually I'm thinking... We have a 
prefect-flows
project Now I have and I want to tell prefect to use the latest version of this project when executing the flows. Maybe I can: 1. Have the flows in prefect-flows project. 2. On feast project build the docker image and call prefect api to tell there is a new image. Prefect will just use the latest configuration one way or another 🤷 In this possible @Anna Geller? Modify the storage "on demand" through the api?
Ok last easiest solution, trigger a new build pipeline from feast project telling prefect flows project to build the flows 🤔
a
Interesting approach! Instead of looping over flows before registration in order to attach storage to each, you may approach it via a shared function. But there is no right or wrong here, the for-loop approach is totally valid, too! It may be worth noting: when you run flow locally, then storage and run configurations are ignored, so attaching it directly to the Flow object is fine and won't result in flow registration when triggering a local run.
How do you deal with distributing the code for the flows?
One idea worth exploring is using flow storage purely for flow code (e.g. using S3 or GitHub) and then packaging code dependencies via a separate mechanism. The most convenient approach is building a custom package since for local development, you can even install that in editable mode if you want, and use the same logic to python code dependencies for production by installing the same package within your Docker image
In this possible @Anna Geller? Modify the storage "on demand" through the api?
Technically, it may be possible since the reference to the storage is stored in the backend as flow metadata (as opposed to e.g. executor which is read from storage at runtime). But we don't expose any hooks to do that and the solution might get convoluted, so it may be easier to avoid that route. But if you want to, I can check the source code and GraphQL API docs to dive deeper btw, I wouldn't focus too much on building best practices for that in 1.0 - this will be easier to accomplish in Prefect 2.0 due to the separation of concerns here - you would define such information separately from flow in a 
DeploymnetSpec
. So I'd be more curious to discuss how to approach this in Prefect 2.0 if you want to chat more
a
Yes I'm just being lazy and delaying the upgrade, but I say that's easier probably I should start there :P
👍 1
Should I move 2.0 then? Btw we have a business account, is the same account valid for both cloud access?
a
If you are on a self-serve tier, you would need to sign up for Cloud 2.0 account here and get started as shown here. For agent/infra setup, we will build more and more examples under this Discourse tag if you are on Enterprise, you can talk to Sales and they have a cool process to make migration easy: sales@prefect.io
and yes, I would encourage you to start migrating some workflows already! I think doing a big bang migration is hard, so IMO it's easier to start already and migrate perhaps some simpler/less critical workflows to Prefect 2.0 and Cloud 2.0 - this topic dives deeper
a
amazing, thank you. @Anna Geller
👍 1
interesting it seems is possible to migrate flows individually
👍 1
💯 1
38 Views
Open in Slack
PreviousNext
Powered by