If you want to keep your existing code base as-is without adding a Prefect dependency, might it be useful to import your existing functions and run them in Prefect flows and tasks?
For example:
Copy code
from prefect import flow, task
from your_module import preprocessing_function
from your_other_module import training_function
@task
def preprocess_data():
preprocessing_function()
@task
def train():
training_function()
@flow(name='My pipeline')
def process_and_train():
preprocess_data()
train()
This would let you introduce Prefect gradually without needing to alter your existing code - so anyone who wants to keep using the code base as-is can do so, and anyone who wants to use the Prefect flows you are building can install Prefect.
upvote 2
o
Ofir
08/26/2022, 2:18 PM
Thanks @Ryan Peden I appreciate that.
I had exactly this plan in mind but was wondering whether it’s the way to go
Ofir
08/26/2022, 2:19 PM
The only downside is the amount of boilerplate and maintenance, it means that every change in my code (flow) structure needs to be accompanied by a change in Prefect
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.