Thread
#prefect-community
    g

    Giang Hoang Le

    2 years ago
    OOP (polymorphism) in Prefect Hi Guys, I would like to ask is there any OOP method in Prefect Framework. I would like to create a base structure for all various methods in different sources. So I would like to ask is this possible to do in Prefect framework? Kind Regards Giang
    Jeremiah

    Jeremiah

    2 years ago
    Hi @Giang Hoang Le, Prefect is a fully class-based system, though most of our examples use a functional API. You can take any
    Task
    class and subclass it with normal Python inheritance.
    The
    @task
    decorator and most examples that call tasks like functions show a “functional API” but the “imperative API” embraces the full class-based set up (https://docs.prefect.io/core/getting_started/first-steps.html#imperative-api).
    g

    Giang Hoang Le

    2 years ago
    I would like to ask if I have different functions in a class. Could I use them when I build a Flow and serialize the flow to a cluster to excuse?
    like: source = Source(source_name) with Flow('S3 to Redshift-s3😕/bankdata/2019/*.csv') as flow: result_discorver = source.discover() result_extract = source.extract() result transform = source.transform() result_load = source.load() result_verify = source.verify() flow.deploy()
    Thank Jeremiah!
    Jeremiah

    Jeremiah

    2 years ago
    Ah, I understand. That’s a fantastic question - if your methods were decorated with
    @staticmethod and @task
    then I’m sure it would work. If they are
    classmethods
    or “normal” methods then the first argument might create an issue
    I’m actually not sure of the behavior if you decorated a “normal” method with
    @task
    g

    Giang Hoang Le

    2 years ago
    I would like to implement all methods in class Source(Task) maybe and vary it depend on source_name.
    I will try staticmethod one
    Jeremiah

    Jeremiah

    2 years ago
    Another option is for each method to create new
    Task
    objects, call them, and return them, something like:
    class Source:
        def transform(self, x):
            # some logic that depends on Source
            task = MyTransformTaskClass()
            return task(x)
    I think it depends on where you want to encapsulate the task logic
    g

    Giang Hoang Le

    2 years ago
    Ooh, Okay that is cool!!!
    Jeremiah

    Jeremiah

    2 years ago
    In general, the
    Task
    itself is a full class so it will be difficult for it to exist exclusively as a method of another class. However,
    @staticmethod
    +
    @task
    is one way to get there (but, because its a staticmethod, you won’t be able to access the parent
    Source)
    So I’m wondering instead if you use
    Source
    to parameterize your “factory methods” that generate and return tasks dynamically depending on how
    Source
    was initialized
    Please let us know how you decide to do this
    I’m very interested
    g

    Giang Hoang Le

    2 years ago
    Jeremiah, my idea is just logging a warn if subclass has not implement some methods, it miss from Source class and reuse same methods in Source class for much more convenient 😁, Thanks a lot!