https://prefect.io logo
Title
r

Rui Rego

02/04/2021, 2:57 PM
Hi. I have a flow that passes a results object (the results of session.query - sqlalchemy) between tasks. If I run the flow with flow.run() it executes ok. If I run it from the UI it fails with this error:
Unexpected error: TypeError("cannot pickle 'weakref' object")
Traceback (most recent call last):
  File "d:\302_investing\venv\lib\site-packages\prefect\engine\runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "d:\302_investing\venv\lib\site-packages\prefect\engine\task_runner.py", line 891, in get_task_run_state
    result = self.result.write(value, **formatting_kwargs)
  File "d:\302_investing\venv\lib\site-packages\prefect\engine\results\local_result.py", line 116, in write
    value = self.serializer.serialize(new.value)
  File "d:\302_investing\venv\lib\site-packages\prefect\engine\serializers.py", line 70, in serialize
    return cloudpickle.dumps(value)
  File "d:\302_investing\venv\lib\site-packages\cloudpickle\cloudpickle_fast.py", line 73, in dumps
    cp.dump(obj)
  File "d:\302_investing\venv\lib\site-packages\cloudpickle\cloudpickle_fast.py", line 563, in dump
    return Pickler.dump(self, obj)
TypeError: cannot pickle 'weakref' object
Why the different behavior between Core and Server?
âś… 1
c

Chris White

02/04/2021, 3:47 PM
Hi @Rui Rego - when using a Prefect backend like Server, task results are checkpointed by default, which requires the results to be serializable. This doc discusses the configuration of checkpointing (including the ability to turn it off): https://docs.prefect.io/core/concepts/results.html
r

Rui Rego

02/05/2021, 1:32 AM
Thanks. On this particular flow I've changed the task decorators to:
@task(max_retries=5, retry_delay=datetime.timedelta(seconds=5), state_handlers=[alert_failed], *checkpoint=False*)
It now executes without the error above.
đź’Ż 1
c

Chris White

02/05/2021, 1:35 AM
@Marvin archive “Different checkpoint behavior between Core and Server”