David Harrington
03/03/2021, 7:41 PMPREFECT__FLOWS__CHECKPOINTING: false
for the flow. Based on the logs/errors we are seeing - the flow is still trying to serialize and write results. Two questions:
1. if we disable checkpointing at the flow level should prefect core not attempt to serialize the result
2. what is the appropriate way to disable checkpointing at the flow level
Relevant traceback included as a comment from when we ran a job where the env var was set to false[2021-03-03 17:53:38+0000] ERROR - prefect.CloudTaskRunner | Unexpected error: TypeError("can't pickle _thread.lock objects")
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/prefect/engine/task_runner.py", line 891, in get_task_run_state
result = self.result.write(value, **formatting_kwargs)
File "/usr/local/lib/python3.7/dist-packages/prefect/engine/results/s3_result.py", line 100, in write
binary_data = new.serializer.serialize(new.value)
Jim Crist-Harif
03/03/2021, 8:09 PMS3Result
yourself, and its instead set automatically from S3
storage?David Harrington
03/03/2021, 8:09 PMJim Crist-Harif
03/03/2021, 8:10 PMDavid Harrington
03/03/2021, 8:10 PMJim Crist-Harif
03/03/2021, 8:10 PMMarvin
03/03/2021, 8:10 PMJim Crist-Harif
03/03/2021, 8:11 PM{"checkpointing": False}
in the context for a flow run, that should work.PREFECT__CONTEXT__CHECKPOINTING=false
might also workDavid Harrington
03/03/2021, 8:12 PMJim Crist-Harif
03/03/2021, 8:48 PM