David Martin Calalang
07/14/2025, 2:16 PMNate
07/14/2025, 2:19 PMNate
07/14/2025, 2:19 PMNate
07/14/2025, 2:19 PMDavid Martin Calalang
07/14/2025, 2:25 PMNate
07/14/2025, 2:33 PMwith_options
? or how are you doing this?
I'm trying to pass it in later on
David Martin Calalang
07/14/2025, 2:35 PMwith_options
as well as flow.task_runner=___
David Martin Calalang
07/14/2025, 2:36 PMNate
07/14/2025, 2:37 PMflow.task_runner=___
that is what with_options
is for
do you have an example where its not using the selected task runner reliably?Nate
07/14/2025, 2:38 PMDavid Martin Calalang
07/14/2025, 3:29 PMDavid Martin Calalang
07/14/2025, 3:29 PMsquare
is the name of the task that I'm passing to my flow...
Traceback (most recent call last):
File "/usr/local/lib/python3.13/site-packages/distributed/protocol/pickle.py", line 60, in dumps
result = pickle.dumps(x, **dump_kwargs)
_pickle.PicklingError: Can't pickle <function square at 0x7b68d25cd1c0>: it's not the same object as __main__.square
For more context, I'm using the dask task_runner with a pre-existing Fargate cluster I have on AWS ECS. From my understanding prefect serializes tasks prior to sending them to the dask scheduler, and that might be where my issue is. Not quite sure where to look/what to fix in my code regarding this though.
Thanks!