Prasanth Kothuri
04/12/2022, 4:31 PMFile "/usr/local/lib/python3.8/dist-packages/cloudpickle/cloudpickle_fast.py", line 73, in dumps
cp.dump(obj)
File "/usr/local/lib/python3.8/dist-packages/cloudpickle/cloudpickle_fast.py", line 602, in dump
return Pickler.dump(self, obj)
TypeError: cannot pickle '_thread.lock' object
to make it simple it has just one task as below
@task(log_stdout=True)
def get_file_names():
files = s3.Bucket(s3_bucket).objects.all()
file_names = []
for my_bucket_object in files:
file_name = my_bucket_object.key
regex = re.search(r".ctl", str(file_name))
if regex is not None:
file_names.append(file_name)
return file_names
and flow
# flow to chain the tasks
with Flow("my_flow", storage=storage, schedule=schedule) as f:
ctl_files = get_file_names()
any ideas why prefect is unable to serialize / pickle ???Anna Geller
04/12/2022, 4:34 PM