Hello, I’m having this error, all the solutions seems outdated
Copy code
TypeError: cannot pickle '_thread.lock' object
a
Anna Geller
10/08/2021, 1:08 PM
Can you share more info about your flow structure? Do you use any database connections or HTTP clients? Those cannot be serialized and sent between Dask workers. If you don’t use Dask, you may look at Script-based storage. Kevin explained it here.
upvote 2
k
Kevin Kho
10/08/2021, 1:43 PM
Hey @Joseph Oladokun, this is likely from a task having an output that can’t be serialized as Anna mentioned
j
Joseph Oladokun
10/12/2021, 11:57 AM
Thanks @Anna Geller for your response, yes, i use http clients to fetch data from an API
Joseph Oladokun
10/12/2021, 11:58 AM
@Kevin Kho I’m fetching data from an API and writing it to the s3 bucket
a
Anna Geller
10/12/2021, 11:59 AM
Did you manage to solve it? If you move the boto3 client to the task in which you interact with S3, this should solve the serialization issue.
j
Joseph Oladokun
10/12/2021, 12:34 PM
Thanks @Anna Geller, I have been able to solve the issue