https://prefect.io logo
c

Chirag

10/13/2020, 10:59 AM
Hi Guys, Is there a way to supply MySQL connection to multiple tasks? Currently, I am getting this error : "TypeError: can't pickle _mysql_connector.MySQL objects"
a

ale

10/13/2020, 11:32 AM
Afaik the best practice is to pass connection information like dbname, host, user, pwd directly to teh MySQLTask. Passing the connector would cause Prefect to not being able to serialize your flow, hence the pickling issue
c

Chirag

10/13/2020, 11:55 AM
I tried importing MySQLFetch using: prefect.tasks.mysql.mysql.MySQLFetch, but it is saying no such module exists. I have installed prefect with all_extras.
a

ale

10/13/2020, 12:01 PM
Strange. If you can share the code that is not working maybe someone from the community can help you figuring it out
j

Jim Crist-Harif

10/13/2020, 1:11 PM
That module was moved in a recent prefect release. You can try upgrading, or importing from
prefect.contrib....
And yeah, as @ale said, this has to do with connection objects not being pickleable. We recommend creating a new connection in each task. There's some ongoing work in core to make this more efficent, so tasks running in the same process can share a connection between them, but that work hasn't been completed yet.
c

Chirag

10/14/2020, 9:27 AM
I have created new connections in each of the tasks and it worked. Thanks @Jim Crist-Harif and @ale
👍 1
3 Views