OOP polymorphism in Prefect Hi Guys I would like to ask is Prefect Community #ask-community

OOP (polymorphism) in Prefect Hi Guys, I would li...

Giang Hoang Le

12/10/2019, 11:03 PM

OOP (polymorphism) in Prefect Hi Guys, I would like to ask is there any OOP method in Prefect Framework. I would like to create a base structure for all various methods in different sources. So I would like to ask is this possible to do in Prefect framework? Kind Regards Giang

Jeremiah

12/10/2019, 11:05 PM

Hi @Giang Hoang Le, Prefect is a fully class-based system, though most of our examples use a functional API. You can take any

Task

class and subclass it with normal Python inheritance.

Jeremiah

12/10/2019, 11:06 PM

The

@task

decorator and most examples that call tasks like functions show a “functional API” but the “imperative API” embraces the full class-based set up (https://docs.prefect.io/core/getting_started/first-steps.html#imperative-api).

Giang Hoang Le

12/10/2019, 11:08 PM

I would like to ask if I have different functions in a class. Could I use them when I build a Flow and serialize the flow to a cluster to excuse?

Giang Hoang Le

12/10/2019, 11:08 PM

like: source = Source(source_name) with Flow('S3 to <Redshift-s3://bank>_data/2019/*.csv') as flow: result_discorver = source.discover() result_extract = source.extract() result_ transform = source.transform() result_load = source.load() result_verify = source.verify() flow.deploy()

Giang Hoang Le

12/10/2019, 11:32 PM

Thank Jeremiah!

Jeremiah

12/10/2019, 11:34 PM

Ah, I understand. That’s a fantastic question - if your methods were decorated with

@staticmethod and @task

then I’m sure it would work. If they are

classmethods

or “normal” methods then the first argument might create an issue

Jeremiah

12/10/2019, 11:35 PM

I’m actually not sure of the behavior if you decorated a “normal” method with

@task

Giang Hoang Le

12/10/2019, 11:36 PM

I would like to implement all methods in class Source(Task) maybe and vary it depend on source_name.

Giang Hoang Le

12/10/2019, 11:37 PM

I will try staticmethod one

Jeremiah

12/10/2019, 11:38 PM

Another option is for each method to create new

Task

objects, call them, and return them, something like:

Jeremiah

12/10/2019, 11:38 PM

Copy code

class Source:
    def transform(self, x):
        # some logic that depends on Source
        task = MyTransformTaskClass()
        return task(x)

👍 1

Jeremiah

12/10/2019, 11:39 PM

I think it depends on where you want to encapsulate the task logic

Giang Hoang Le

12/10/2019, 11:40 PM

Ooh, Okay that is cool!!!

Jeremiah

12/10/2019, 11:40 PM

In general, the

Task

itself is a full class so it will be difficult for it to exist exclusively as a method of another class. However,

@staticmethod

@task

is one way to get there (but, because its a staticmethod, you won’t be able to access the parent

Source)

Jeremiah

12/10/2019, 11:41 PM

So I’m wondering instead if you use

Source

to parameterize your “factory methods” that generate and return tasks dynamically depending on how

Source

was initialized

Jeremiah

12/10/2019, 11:41 PM

Please let us know how you decide to do this

👍 1

Jeremiah

12/10/2019, 11:41 PM

I’m very interested

Giang Hoang Le

12/11/2019, 12:34 AM

Jeremiah, my idea is just logging a warn if subclass has not implement some methods, it miss from Source class and reuse same methods in Source class for much more convenient 😁, Thanks a lot!

3 Views

Open in Slack

Previous Next