https://prefect.io logo
Powered by
Title
g

Giang Hoang Le

12/10/2019, 11:03 PM
OOP (polymorphism) in Prefect Hi Guys, I would like to ask is there any OOP method in Prefect Framework. I would like to create a base structure for all various methods in different sources. So I would like to ask is this possible to do in Prefect framework? Kind Regards Giang
j

Jeremiah

12/10/2019, 11:05 PM
Hi @Giang Hoang Le, Prefect is a fully class-based system, though most of our examples use a functional API. You can take any 
Task
class and subclass it with normal Python inheritance.
The 
@task
decorator and most examples that call tasks like functions show a “functional API” but the “imperative API” embraces the full class-based set up (https://docs.prefect.io/core/getting_started/first-steps.html#imperative-api).
g

Giang Hoang Le

12/10/2019, 11:08 PM
I would like to ask if I have different functions in a class. Could I use them when I build a Flow and serialize the flow to a cluster to excuse?
like: source = Source(source_name) with Flow('S3 to <Redshift-s3://bank>_data/2019/*.csv') as flow: result_discorver = source.discover() result_extract = source.extract() result_ transform = source.transform() result_load = source.load() result_verify = source.verify() flow.deploy()
Thank Jeremiah!
j

Jeremiah

12/10/2019, 11:34 PM
Ah, I understand. That’s a fantastic question - if your methods were decorated with 
@staticmethod and @task
then I’m sure it would work. If they are 
classmethods
or “normal” methods then the first argument might create an issue
I’m actually not sure of the behavior if you decorated a “normal” method with 
@task
g

Giang Hoang Le

12/10/2019, 11:36 PM
I would like to implement all methods in class Source(Task) maybe and vary it depend on source_name.
I will try staticmethod one
j

Jeremiah

12/10/2019, 11:38 PM
Another option is for each method to create new 
Task
objects, call them, and return them, something like:
class Source:
    def transform(self, x):
        # some logic that depends on Source
        task = MyTransformTaskClass()
        return task(x)
👍 1
I think it depends on where you want to encapsulate the task logic
g

Giang Hoang Le

12/10/2019, 11:40 PM
Ooh, Okay that is cool!!!
j

Jeremiah

12/10/2019, 11:40 PM
In general, the 
Task
itself is a full class so it will be difficult for it to exist exclusively as a method of another class. However, 
@staticmethod
+ 
@task
is one way to get there (but, because its a staticmethod, you won’t be able to access the parent 
Source)
So I’m wondering instead if you use 
Source
to parameterize your “factory methods” that generate and return tasks dynamically depending on how 
Source
was initialized
Please let us know how you decide to do this
👍 1
I’m very interested
g

Giang Hoang Le

12/11/2019, 12:34 AM
Jeremiah, my idea is just logging a warn if subclass has not implement some methods, it miss from Source class and reuse same methods in Source class for much more convenient 😁, Thanks a lot!
#prefect-communityJoin Slack
Powered by