https://prefect.io logo
Powered by
Title
d

David Harrington

03/03/2021, 7:41 PM
Hi we are running flows via a k8s agent and we are trying to disable checkpointing. We are using a k8s yaml template that sets the environment variable 
PREFECT__FLOWS__CHECKPOINTING: false
for the flow. Based on the logs/errors we are seeing - the flow is still trying to serialize and write results. Two questions: 1. if we disable checkpointing at the flow level should prefect core not attempt to serialize the result 2. what is the appropriate way to disable checkpointing at the flow level Relevant traceback included as a comment from when we ran a job where the env var was set to false
[2021-03-03 17:53:38+0000] ERROR - prefect.CloudTaskRunner | Unexpected error: TypeError("can't pickle _thread.lock objects")
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/prefect/engine/runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/prefect/engine/task_runner.py", line 891, in get_task_run_state
    result = self.result.write(value, **formatting_kwargs)
  File "/usr/local/lib/python3.7/dist-packages/prefect/engine/results/s3_result.py", line 100, in write
    binary_data = new.serializer.serialize(new.value)
j

Jim Crist-Harif

03/03/2021, 8:09 PM
Hmmm, we currently hardcode that to on, but I'm not sure why.
I assume you didn't set the 
S3Result
yourself, and its instead set automatically from 
S3
storage?
d

David Harrington

03/03/2021, 8:09 PM
Yeah exactly
We got around this by setting each task to checkpoint=False
j

Jim Crist-Harif

03/03/2021, 8:10 PM
We should be able to support this though. I'll raise an issue.
d

David Harrington

03/03/2021, 8:10 PM
Awesome, thanks!
j

Jim Crist-Harif

03/03/2021, 8:10 PM
@Marvin open "Support disabling checkpointing at the flow level"
m

Marvin

03/03/2021, 8:10 PM
https://github.com/PrefectHQ/prefect/issues/4201
❤️ 1
j

Jim Crist-Harif

03/03/2021, 8:11 PM
Actually, if you set 
{"checkpointing": False}
in the context for a flow run, that should work.
👍 1
PREFECT__CONTEXT__CHECKPOINTING=false
might also work
👀 1
no promises, but that'd be good to try
d

David Harrington

03/03/2021, 8:12 PM
Interesting thanks for the response - I’m not sure we’re going to get around to testing that soon but we will definitely keep this in mind when we get back to this on our end
j

Jim Crist-Harif

03/03/2021, 8:48 PM
NVM, it's hardcoded in a different spot than I remembered, none of the above will work.
2 Views
#prefect-serverJoin Slack
Powered by