Hi, I try to see if Prefect would be a good fit for our different usecases and I come across a challenge when I try to use Prefect for one of our Pipeline. I have a class called Orchestrator that have an init method and different methods. One of the method is the main process that call another class. My initial ideas was to put directly flow and task on each of my function that I would like to monitor but I saw that it’s not working. I have the impression that I cannot use flow and task directly inside my class. It’s a problem for me since I don’t want to modify my pipeline. However, I try to implement it by extracting outside of my class three big steps: let’s say, download(), transform(), load() It works. however, my transform method call others function of my class that I would like to monitor. load call a big process() function on another class which contain multiple substep function. I don’t know how can I monitor properly. It would be painful If need to find a way to extract each function outside of the class since I will loose some benefits. Any insights ? Is there something I missed ?

👋 @Lirone! So you’re looking for something like:

class Orchestrator:

    @task
    def extract():
 
    @task 
    def transform():

    @task
    def load():

    @flow
    def ETL():
        load(extract(transform()))

?

Yes someting like that except that all my function have self

yep yep was leaving out self for brevity

example.py

Yep! But it takes a little effort to basically make @flow and @task into classmethods. I had something working yesterday and can share when I’m back at a computer.

👀 I came across the same situation myself today with a very similar pipeline setup. Curious to see what you all came up with

Woah good reminder to actually share this woooooof 😅

Still no news @Alexander Azzam ?