https://prefect.io logo
#prefect-community
Title
# prefect-community
h

Hawkar Mahmod

04/05/2022, 10:54 AM
Hey folks, I have a false positive error occurring when a flow runs. It concerns an SQLAlchemy Result object not being pickable. The flow achieves it’s objective successfully, with data inserted into a database but I still get this error. What is Prefect pickling at runtime that could be the cause of this error. Here’s the specific output from the logs:
Unexpected error: TypeError("can't pickle sqlalchemy.cprocessors.UnicodeResultProcessor objects")
1
a

Anna Geller

04/05/2022, 11:03 AM
Can you provide a little more background - is this for Prefect 1.0 or 2.0? Are you on Cloud or self-hosted? What storage do you use and how do you define storage, run config and results in your flow?
h

Hawkar Mahmod

04/05/2022, 12:10 PM
Hey Anna, thanks for the question. I wasn’t sure what would be relevant. • Prefect Core version: 0.15.6 • Prefect Cloud • ECS RunConfig (configured for FarGate) • S3 Storage All storage and RunConfig are defined at flow build time in the same module.
a

Anna Geller

04/05/2022, 12:34 PM
Thanks! Are you using sqlalchemy within your flow and do you happen to have the DB connection defined within your Flow's scope? Sharing your flow code would help to say why you are getting this pickling error - you can share via DM for privacy or redact any sensitive info and you can post it here
k

Kevin Kho

04/05/2022, 2:48 PM
This looks like it’s pickling the return of your task and trying to store it as a result, you can turn off the default checkpointing, or return something else
h

Hawkar Mahmod

05/17/2022, 8:02 AM
Apologies for the very late response to this. @Kevin Kho I switched the checkpointing off and it now works. The result of the insert was an SQLAlchemy object which couldn’t be pickled. I didn’t notice it when developing locally because I don’t think the checkpointing and result persistence operates when just using
flow.run()
.
a

Anna Geller

05/17/2022, 12:55 PM
You're right, result persistence is by default only turned on for flows running on the backend, not for local flow runs. nice you figured that out!
h

Hawkar Mahmod

05/18/2022, 1:43 PM
Thanks @Anna Geller. I suppose it’s best to run the flow using the CLI perhaps rather than
flow.run()
to get as close to production as possible.
a

Anna Geller

05/18/2022, 2:50 PM
100% yes, CLI is best and allows for cleaner flow code
3 Views