Thread
#prefect-community
    s

    Saranya Elumalai

    2 years ago
    I have retry for my task as follows
    @task(max_retries=3, retry_delay=timedelta(seconds=5))
    def write_delta_lookup_table():
    ......
    My image build failed for the reason
    ValueError: Task <Task: write_delta_lookup_table> has retry settings but some upstream dependencies do not have result types. See <https://docs.prefect.io/core/concepts/results.html> for more details.
    Why the task retry require result types. I am not passing values from one task to another. Also https://docs.prefect.io/core/concepts/tasks.html#overview retry example doesnt show the use of result type. What am I missing here. ?? (Any documentation with simple retry usecase will be very helpful)
    Chris White

    Chris White

    2 years ago
    Hi @Saranya Elumalai - what version of Prefect are you running on? I believe we removed this check recently. Regardless, if you add:
    from prefect.engine.results import LocalResult
    
    # when you initialize your Flow
    Flow(..., result=LocalResult(), ...)
    to your Flow initialization, this check will not fail
    s

    Saranya Elumalai

    2 years ago
    @Chris White Using prefect core 0.11.2. Does this version still check?
    Chris White

    Chris White

    2 years ago
    If youโ€™re running into the error it must - the patch I provided above or upgrading your Core version should resolve the issue for you
    s

    Saranya Elumalai

    2 years ago
    thanks, let me try localresult and update you
    @Chris White The build is successful, but the task failed with the following error
    Unexpected error: TypeError("can't pickle SSLContext objects")
    
     Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
        new_state = method(self, state, *args, **kwargs)
      File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 977, in get_task_run_state
        result = self.result.write(value, filename="output", **prefect.context)
    Here is my code
    with Flow(f"store-etl",result=LocalResult()) as flow:
    company = write_delta_lookup_table()
    .......
    @task(max_retries=3, retry_delay=timedelta(seconds=10))
    def get_S3_connection():
    What am I missing here
    Chris White

    Chris White

    2 years ago
    that code looks like an old version of Prefect - you should confirm that your Docker image also has the newest version of Prefect installed. Regardless, tasks must return objects that are capable of being serialized by
    cloudpickle
    , else there is no way to persist them to disk. I see that you are creating an S3 client; for situations like this we have a WIP for shareable non-pickelable objects here: https://github.com/PrefectHQ/prefect/pull/3139
    s

    Saranya Elumalai

    2 years ago
    @Chris White 1. Which code looks like old version 2. Both tasks get_S3_connection() and write_S3_data() facing the same issue when added retry. What are the recommended task return types?
    Chris White

    Chris White

    2 years ago
    The traceback looks like it came from an old version of prefect; anything that is capable of being serialized by cloudpickle is a valid return type - most python objects satisfy this. Database connections and client objects, or anything that maintains a reference to such objects, are generally not serializable
    Anish Chhaparwal

    Anish Chhaparwal

    1 year ago
    hey @Saranya Elumalai and @Chris White i'm facing a similar issue. i have a function that returns the client object of mongodb. I am unable to pass it to other tasks. is there any work around?
    Chris White

    Chris White

    1 year ago
    Hi @Anish Chhaparwal at the moment I suggest you recreate the client within each task to avoid this issue. This PR: https://github.com/PrefectHQ/prefect/pull/3139 will ultimately provide a better pattern for these sorts of situations in the future!
    s

    Saranya Elumalai

    1 year ago
    Yes @Chris White @Anish Chhaparwal thats what I did
    Anish Chhaparwal

    Anish Chhaparwal

    1 year ago
    okay. Thanks for the quick response. appreciate it. ๐Ÿ˜ƒ