Hello We have an issue with prefect dask where if our tasks

Question

Hello We have an issue with prefect dask where if our tasks rely on a complex object being returned e g one that cannot be pickled we have a problem moving from the default `ConcurrentTaskRunner` to the `DaskTaskRunner` Basically we encounter the issue mentioned here <https discourse prefect io t picklingerror error when using a dasktaskrunner 908> What s the best way to work around this other than changing the object being returned assuming that s not feasible is there some way to override how serialization is done on it or why does it work with the regular task runner but not with dask

Tom Klein · Answer

although from the error we get it seems it s complaining about the task function itself rather than its arguments ``` File Users redacted Library Caches pypoetry virtualenvs prefect tool nqGlPUR4 py3 9 lib python3 9 site packages distributed protocol pickle py line 63 in dumps result = pickle dumps x dump kwargs pickle PicklingError Can t pickle lt function get feature group data at 0x1334dcee0 gt it s not the same object as main get feature group data```

Tom Klein · Answer

some more info from the error itself ```2023 07 09 15 50 48 768 distributed protocol pickle ERROR Failed to serialize task lt prefect tasks Task object at 0x1065e7f70 gt task run TaskRun id=UUID 46e50a13 9074 4ebc 8c9d e7e4dc2eb499 name= Get Feature Group Data 0 flow run id=UUID 94c1a094 2865 47cd 8540 0f9e8146acfd task key= main get feature group data dynamic key= 0 cache key=None cache expiration=None task version=None empirical policy=TaskRunPolicy max retries=0 retry delay seconds=0 0 retries=0 retry delay=0 retry jitter factor=None tags= state id=UUID fdd1635e 3468 45ff a0ab b31ca5491a18 task inputs= feature group TaskRunResult input type= task run id=UUID 676aedf2 f950 4712 94f4 7047fff48f36 state type=StateType PENDING state name= Pending run count=0 flow run run count=0 expected start time=DateTime 2023 7 9 12 50 48 730929 tzinfo=Timezone +00 00 next scheduled start time=None start time=None end time=None total run time=datetime timedelta 0 estimated run time=datetime timedelta 0 estimated start time delta=datetime timedelta microseconds=50067 state=Pending message=None type=PENDING result=None parameters feature group HBFeatureGroup name= accounts context sagemaker session= lt sagemaker session Session object at 0x12c6180d0 gt feature definitions= wait for None result factory ResultFactory persist result=False cache result in memory=True serializer=PickleSerializer type= pickle picklelib= cloudpickle picklelib version= 2 2 1 storage block id=UUID cef80357 3b80 4dcc bbac c554dff38b71 storage block=LocalFileSystem basepath= Users redacted prefect storage storage key fn= lt function DEFAULT STORAGE KEY FN at 0x128d511f0 gt log prints False settings SettingsContext profile=Profile name= default settings= lt PREFECT API KEY str gt redacted ```