Hi everyone has anyone had experience using Dask and faced a Prefect Community #ask-community

Hi everyone, has anyone had experience using Dask ...

David Martin Calalang

08/20/2025, 5:30 PM

Hi everyone, has anyone had experience using Dask and faced a pickling error? Right now my setup is that I have a Prefect work pool with a worker and server both hosted on AWS ECS. But I have one flow that uses Dask and I get the following message

TypeError: ('Could not serialize object of type LLGExpr',

for the Prefect task that I'm submitting to Dask/task runner. I've faced this issue before and didn't manage to find a fix but was just curious if anyone else has faced something similar

Nate

08/20/2025, 5:31 PM

have you checked out this doc?

Nate

08/20/2025, 5:32 PM

some things are genuinely not pickleable but sometimes you can tweak the thing so it serializes differently

Nate

08/20/2025, 5:33 PM

not strictly about caching, but serialization in general

David Martin Calalang

08/20/2025, 5:33 PM

Thanks! and just to check LLGExpr isn't something Prefect specific right? I assumed it was from the upstream (Dask)

Nate

08/20/2025, 5:34 PM

yea that's not a prefect thing

👍 1

Nate

08/20/2025, 5:34 PM

https://github.com/dask/dask/blob/d9b5c5b0256208f1befe94b26bfa8eaabcd0536d/dask/_expr.py#L961

Nate

08/20/2025, 5:35 PM

if your usage seems normal to you, feel free to create an issue or discussion and we can check out whether there's something to improve in

prefect-dask

David Martin Calalang

08/20/2025, 5:36 PM

Sounds good, will try this out. Thanks again!

David Martin Calalang

08/20/2025, 5:40 PM

So after skimming through the docs, I don't think this applies to my case. I didn't include the full error message but

TypeError: ('Could not serialize object of type LLGExpr'

is referring to a function/task that couldn't be serialized, not a task input.

David Martin Calalang

08/20/2025, 5:41 PM

prefect-dask has worked on much simpler cases/tasks, but for some reason when I start using it on more practical use cases I face the same LLGExpr pickling issue

Nate

08/20/2025, 5:41 PM

if you have an MRE for that, please feel free to share

David Martin Calalang

08/20/2025, 5:53 PM

will get back to you on that

🙏 1

David Martin Calalang

08/20/2025, 6:43 PM

I think

David Martin Calalang

08/20/2025, 6:44 PM

lol I was trying to make an MRE (ie. make my script simpler) and I think it ended up working

catjam 1

Nate

08/20/2025, 7:23 PM

that is part of why we ask for MREs 🙂

🔥 1

David Martin Calalang

08/20/2025, 7:28 PM

so my tasks had a failure handler attached to them which was linked to a boto3 library function I made. I think that was the issue since I don't believe aws clients are pickle-able

David Martin Calalang

08/20/2025, 7:29 PM

though I feel like we could get around that with the non-serializable cache page you sent earlier

8 Views

Open in Slack

Previous Next