In Prefect we can define tasks as functions with

@task

decorator, or as classes by subclassing

Task

. In general when programming in python and presented with these options, I typically reach for the subclassing option. I find I can think about problems at a higher level with OOP, easier to build up complex components from simpler ones, cleaner abstraction, etc. However I do find it’s a bit clunky with at times when building flows. Is it recommended to avoid the subclassing approach and use a functional approach instead when possible?

I don’t think Prefect makes recommendations around this, and adapts to whatever you prefer. It is recommended though to use the functional API over imperative API in the flow block.

It seems like going too deeply into an OOP paradigm gets you in trouble though. For instance, implementing logic with class methods isn’t terribly useful because they can’t be called directly as a prefect task. You have to write a wrapper function that takes the object and then calls the methods. But at that point it probably would have been simpler to just implement the logic in a pure function in the first place. I don’t think it’s actually a bad thing to be skew more functional-programming oriented. There seems to be a movement away from OOP and towards a cleaner separation of data and logic within the programming community in general. I guess I just wanted to sanity check if this resonated with folks or not, as I start to think about the architecture of my next batch of pipelines.

Ah I see what you mean. I think as long as the class is serializable, you can pass it around (think pandas DataFrame). If you have a class that really holds state of something and is being mutated, you kinda need the wrapper around a class (you would make task wrappers around pandas Dataframes). You are right though that this can’t be used directly. Have you seen the Orion docs teased out last night by chance? https://orion-docs.prefect.io/

@Kevin Kho I haven’t yet but I will definitely go through it today! Ya and you’re right that its really mutability of the object that can become an issue. Maybe designing a workflow around immutable dataclasses and functions is just a more natural fit than designing one around traditional classes. That is kinda conventional distributed programming wisdom now that I think about it. If you have to serialize and pass around the object anyway, seems it’s best to only pass around the data and keep the logic completely separate in functions that can be made directly into tasks. Will give it a try on the next batch of pipelines I work on.

That makes sense and because of how coupled Prefect is to Dask, I think Prefect’s design encourages the tasks to be parallelizeable