Hi we are running flows via a k8s agent and we are...
# prefect-server
d
Hi we are running flows via a k8s agent and we are trying to disable checkpointing. We are using a k8s yaml template that sets the environment variable
PREFECT__FLOWS__CHECKPOINTING: false
for the flow. Based on the logs/errors we are seeing - the flow is still trying to serialize and write results. Two questions: 1. if we disable checkpointing at the flow level should prefect core not attempt to serialize the result 2. what is the appropriate way to disable checkpointing at the flow level Relevant traceback included as a comment from when we ran a job where the env var was set to false
Copy code
[2021-03-03 17:53:38+0000] ERROR - prefect.CloudTaskRunner | Unexpected error: TypeError("can't pickle _thread.lock objects")
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/prefect/engine/runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/prefect/engine/task_runner.py", line 891, in get_task_run_state
    result = self.result.write(value, **formatting_kwargs)
  File "/usr/local/lib/python3.7/dist-packages/prefect/engine/results/s3_result.py", line 100, in write
    binary_data = new.serializer.serialize(new.value)
j
Hmmm, we currently hardcode that to on, but I'm not sure why.
I assume you didn't set the
S3Result
yourself, and its instead set automatically from
S3
storage?
d
Yeah exactly
We got around this by setting each task to checkpoint=False
j
We should be able to support this though. I'll raise an issue.
d
Awesome, thanks!
j
@Marvin open "Support disabling checkpointing at the flow level"
m
j
Actually, if you set
{"checkpointing": False}
in the context for a flow run, that should work.
👍 1
PREFECT__CONTEXT__CHECKPOINTING=false
might also work
👀 1
no promises, but that'd be good to try
d
Interesting thanks for the response - I’m not sure we’re going to get around to testing that soon but we will definitely keep this in mind when we get back to this on our end
j
NVM, it's hardcoded in a different spot than I remembered, none of the above will work.