We get a pickle thread lock error in a task when using paramiko. This is probably a problem with pickling ssh stuff. What is the alternative? How are people using paramiko with Prefect?
06/22/2022, 1:10 PM
Hi Russell, the answer is here but the gist is you can instantiate clients inside tasks to avoid it or turn off checkpointing for the task. Are you returning some kind of Client?
06/22/2022, 8:17 PM
In my tasks I create the paramiko SSH conn directly in the task. I had a similar issue with sqlalchemy database engines though, and ended up sending only the connection string as the result. For your situation, maybe you could just set the results to a kwargs dictionary, and then use those kwargs in your next task like the below.
06/23/2022, 12:14 AM
Thank you. I will try these.
Putting it inside the task where it is also used worked. Couldn't get it to work passing as a param from another task though, so it is inefficient in that it recreates the paramiko connection for each use… is there a better way in Prefect 2.0? (I am in 1.2.2)
Also thanks again for the prompt and useful replies
06/23/2022, 10:37 PM
You can actually pass it if you turn off checkpointing and aren’t using Dask