Hey folks, I was wondering if it's possible to make our tasks and flows functions as methods of a class, without having the linter screaming at me (and without disabling type checking). Example of what I mean, try this:

class MyClass:
    @task
    def doStuffWithArgs(self, myArg: str):
        ...

    def doStuffWithoutArgs(self):
        ...

    @flow
    def doMoreStuff(self):
        self.doStuffWithArgs("stuff")
        self.doStuffWithoutArgs()

You will see that for

doStuffWithArgs

Pylance complains that no overload matches that call (see screenshot):

in my opinion this design is not easy to maintain anyway

Thanks for the feedback, could you elaborate on that? The reason I wanted a class is because I have certain objects that I want to reuse across different tasks. In my head it made more sense to be able to do

<http://this.my|this.my>_object

instead of either passing it as an arg of each task or making it global

the issue is the args must be pickable

uh dayum you're right

passing the args is probably the best way to work with prefect

what if the arg itself ends up not being pickleable, but the result can? Such as, idk, a duckdb connection object

I believe tasks will not even run without validate_input kwargs in flow decorator

ah, fair enough

the less contextual objects tasks have the better they are 😅

pardon my ignorance, i'm not familiar with the term "contextual object", what do you mean with that ?

tasks should be idempotent

tasks should be idempotent

this I'm aware

by contextual object I would mean global/shared object

uhm ok I understand, altho I'm not sure in this case what would be the difference between a singleton session vs a global shared session object. it's not like im going to initialize it more than once

you could also a global shared session object

seems fine by me, i don't see any drawbacks

mostly DX issues

yeah, just that, but it's a personal project so im not too worried about it

like typing, autcomplete, ect

yup 👍

and if you want reuse it you juste have to implete singleton

Since you're here, I have one last question if you don't mind: Is it possible to cache a task that returns nothing? E.g. imagine a task writes its results to some external storage, and I want it to skip execution if the passed args are the same. Does it make sense ?

you can, and prefect already implement it

task(cache_key_fn=task_input_hash, cache_expiration=None)

I assume you mean this?

but I'm wondering if you will not need a storage anyway

I'm mostly playing around. The idea behind my last question is as follows: Imagine im reading data from some API that returns unpredictable results, and I'm writing them to S3. I don't want duplicate results so if the

write_to_s3

function receives the same input (e.g. result_key), skip execution.

not sure to understand

sorry, lemme explain better

like of the api is a function f, f(5) will alway return the same result ?

like of the api is a function f, f(5) will alway return the same result ?

It should in theory, but in practice it doesn't because of unavailability of the API itself. f(5) sometimes can get 1000 results, sometimes can return only 100. And this 100 can be a subset of the 1000. Each result however has a unique key. So imagine I have two tasks:

get_results_from_api

--->

write_result_to_storage

For each result in

get_result_from_api

, write to storage ONLY if the

result_key

hasn't been seen before.

cache are mostly for retries stuff

ok i understand. I have no idea what "belt and suspender" is tho 😅

prefect cache + checks in the write tasks to ensure the job was not did in the past

ahaha, is this a programming pattern or something you just made up 😄 ?

image.png

got it 👍 , thanks !

np