Will decorating class constructors as task and a function that pipe those tasks as flow work with prefect 2.0
In cases where we create deployments
What is your opinion and what do you suggest?
Should a function be created where that constructor is called and decorating that with flow works better than the previous approach???
a
Anna Geller
05/25/2022, 12:22 PM
Can you explain the problem you are trying to solve?
We have a data pipeline written as a class with constructors for its steps which inherits from Abstract Base class
Above I proposed an approach to wrap those constructors with @task
The same class for that data product has a constructor that connects those steps which can be decorated by a flow so that we can run it as a prefect flow instead of extracting those constructors as functions and creating a flow from it
Kayvan Shah
05/25/2022, 12:51 PM
The constructor solely written for testing purpose
The data product class initialises an abstract class which brings in some required properties to enable it to execute the pipeline
k
Kevin Kho
05/25/2022, 2:27 PM
You are saying:
Copy code
class Test:
@task
def __init__(self, x):
self.x = x
right?
I don’t think this will work. I think you need to make functions that use the class instead and make those into tasks. I am not 100% sure though, you can try it, but I think this will have issues with concurrency
k
Kayvan Shah
05/26/2022, 6:56 AM
constructors other than init
k
Kevin Kho
05/26/2022, 3:55 PM
I still think any method wrapped may cause issues. Double checking with the team tho
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.