https://prefect.io logo
#prefect-community
Title
# prefect-community
p

Prasanth Kothuri

04/12/2022, 4:31 PM
Hello, I am receiving the following pickle error while registering my flow
Copy code
File "/usr/local/lib/python3.8/dist-packages/cloudpickle/cloudpickle_fast.py", line 73, in dumps
    cp.dump(obj)
  File "/usr/local/lib/python3.8/dist-packages/cloudpickle/cloudpickle_fast.py", line 602, in dump
    return Pickler.dump(self, obj)
TypeError: cannot pickle '_thread.lock' object
to make it simple it has just one task as below
Copy code
@task(log_stdout=True)
def get_file_names():
    files = s3.Bucket(s3_bucket).objects.all()
    file_names = []
    for my_bucket_object in files:
        file_name = my_bucket_object.key
        regex = re.search(r".ctl", str(file_name))

        if regex is not None:
            file_names.append(file_name)

    return file_names
and flow
Copy code
# flow to chain the tasks
with Flow("my_flow", storage=storage, schedule=schedule) as f:
    ctl_files = get_file_names()
any ideas why prefect is unable to serialize / pickle ???
a

Anna Geller

04/12/2022, 4:34 PM
Yup, you likely use some not serializable object in your flow code 🙂 Perhaps you can switch to Script storage instead? this would make things easier.
Assuming you are using e.g. S3 storage with a local agent, here is how you could define this: https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/s3_local_run.py#L14
10 Views