Hi, I'm using prefect.context as a di container. • I have an api client thats been created automatically using openapi/swagger • At run time, I store the client n the prefect.context. I can swap it out for stubs etc when running locally I receive the following error though when running: <snip massive stack trace>

TypeError: can't pickle _thread.RLock objects

I've omitted the stack trace for now, as I wanted some guidance on whether I'm using the context incorrectly. I understand that my client is going to be serialized when using Dask. When using the local executor all is fine. Is this a problem with the Client not being serialized correctly or a limitation of prefect? thanks!

Hey @Ralph Willgoss, The error you’re encountering sounds like you’re passing around an unserializable object with Dask. A limitation of distributed computing is the inability to share a thread across two compute environments, whether that’s two processes or two physical machines. I hope that clarifies your situation a bit!

thanks @Kyle Moon-Wright for this. I am aware of the limitation and wanted to double check it was the case re: being the Client. I can further look into why its not serializable. Is my use case of the context ok though?

Out of curiosity, how is your client autogenerated?

@David Ojeda good question, its built as part of a Spring Boot app that it belongs to. It uses the tool https://github.com/swagger-api/swagger-codegen

Oh ok, it must be somewhat like openapi-generator then. We use a rest API on our tasks and we also got that error a while back because the generated client uses threading locks for some internal purposes. What we ended up doing is to setup a prefect secret with the URL and api key of the API, with a helper function that creates a new client from the prefect secret. That way, the information needed to create a client is available and serializable

@Ralph Willgoss prefect context should be considered read-only, as it is essentially “refreshed” for each individual task run

@David Ojeda yeah - I see what you mean. This maybe the path I take too, just exploring ideas. @Chris White i'm using the context as a "write once", when the flow starts. Initialising the object, that all the other tasks will need later on, so its all setup. I don't plan to write back to it once its set, sound ok?

Ah yea providing values to context at the flow run level is supported so that sounds good

Hi, hope it's ok for me to ask this here. Did you ever figure out what parts of the object were unserializable? I'm getting the same error in a different context, and am trying to figure out which part of the object is a problem.

Hi @tkanas, apologies for the delay - I haven't been on here in a while. Yes there was some internal objects to the client that could not be serialized. It was just simple to make our own client, rather than relying on the swagger ones. They got in the way more than they helped in in the end.