https://prefect.io logo
#prefect-community
Title
# prefect-community
r

Ralph Willgoss

05/11/2020, 3:07 PM
Hi, I'm using prefect.context as a di container. • I have an api client thats been created automatically using openapi/swagger • At run time, I store the client n the prefect.context. I can swap it out for stubs etc when running locally I receive the following error though when running: <snip massive stack trace>
TypeError: can't pickle _thread.RLock objects
I've omitted the stack trace for now, as I wanted some guidance on whether I'm using the context incorrectly. I understand that my client is going to be serialized when using Dask. When using the local executor all is fine. Is this a problem with the Client not being serialized correctly or a limitation of prefect? thanks!
k

Kyle Moon-Wright

05/11/2020, 3:29 PM
Hey @Ralph Willgoss, The error you’re encountering sounds like you’re passing around an unserializable object with Dask. A limitation of distributed computing is the inability to share a thread across two compute environments, whether that’s two processes or two physical machines. I hope that clarifies your situation a bit!
r

Ralph Willgoss

05/11/2020, 3:46 PM
thanks @Kyle Moon-Wright for this. I am aware of the limitation and wanted to double check it was the case re: being the Client. I can further look into why its not serializable. Is my use case of the context ok though?
I wondered if the error had something to do with the way prefect serialized things.
d

David Ojeda

05/11/2020, 3:49 PM
Out of curiosity, how is your client autogenerated?
r

Ralph Willgoss

05/11/2020, 3:51 PM
@David Ojeda good question, its built as part of a Spring Boot app that it belongs to. It uses the tool https://github.com/swagger-api/swagger-codegen
d

David Ojeda

05/11/2020, 3:58 PM
Oh ok, it must be somewhat like openapi-generator then. We use a rest API on our tasks and we also got that error a while back because the generated client uses threading locks for some internal purposes. What we ended up doing is to setup a prefect secret with the URL and api key of the API, with a helper function that creates a new client from the prefect secret. That way, the information needed to create a client is available and serializable
📝 1
c

Chris White

05/11/2020, 4:00 PM
@Ralph Willgoss prefect context should be considered read-only, as it is essentially “refreshed” for each individual task run
upvote 1
r

Ralph Willgoss

05/12/2020, 6:12 AM
@David Ojeda yeah - I see what you mean. This maybe the path I take too, just exploring ideas. @Chris White i'm using the context as a "write once", when the flow starts. Initialising the object, that all the other tasks will need later on, so its all setup. I don't plan to write back to it once its set, sound ok?
c

Chris White

05/12/2020, 1:43 PM
Ah yea providing values to context at the flow run level is supported so that sounds good
👍 1
t

tkanas

06/03/2020, 4:08 PM
Hi, hope it's ok for me to ask this here. Did you ever figure out what parts of the object were unserializable? I'm getting the same error in a different context, and am trying to figure out which part of the object is a problem.
r

Ralph Willgoss

09/29/2020, 9:42 AM
Hi @tkanas, apologies for the delay - I haven't been on here in a while. Yes there was some internal objects to the client that could not be serialized. It was just simple to make our own client, rather than relying on the swagger ones. They got in the way more than they helped in in the end.