Do tasks need to be in the same file as the flow which uses

Question

Do tasks need to be in the same file as the flow which uses them thinking face When I had a larger task in a separate module I got PicklingErrors stack trace in thread if I move the task definition into the my main module it works as expected

James Brady · Answer

```Encountered exception during execution Traceback most recent call last File code venv lib python3 10 site packages distributed protocol pickle py line 46 in dumps result = pickle dumps x dump kwargs pickle PicklingError Can t pickle lt function fetch manifests at 0x7f1eee1180d0 gt import of module prefect loader fetch manifests failed During handling of the above exception another exception occurred Traceback most recent call last File code venv lib python3 10 site packages prefect engine py line 557 in orchestrate flow run result = await run sync flow call File code venv lib python3 10 site packages prefect utilities asyncutils py line 56 in run sync in worker thread return await anyio to thread run sync call cancellable=True File code venv lib python3 10 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File code venv lib python3 10 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future File code venv lib python3 10 site packages anyio backends asyncio py line 867 in run result = context run func args File code s2ag main py line 44 in main manifests = fetch manifests submit release File code venv lib python3 10 site packages prefect tasks py line 491 in submit return enter task run engine File code venv lib python3 10 site packages prefect engine py line 687 in enter task run engine return run async from worker thread begin run File code venv lib python3 10 site packages prefect utilities asyncutils py line 136 in run async from worker thread return anyio from thread run call File code venv lib python3 10 site packages anyio from thread py line 49 in run return asynclib run async from thread func args File code venv lib python3 10 site packages anyio backends asyncio py line 970 in run async from thread return f result File usr local lib python3 10 concurrent futures base py line 458 in result return self get result File usr local lib python3 10 concurrent futures base py line 403 in get result raise self exception File code venv lib python3 10 site packages prefect engine py line 792 in create task run then submit future = await submit task run File code venv lib python3 10 site packages prefect engine py line 858 in submit task run future = await task runner submit File code venv lib python3 10 site packages prefect dask task runners py line 191 in submit self dask futures run key = self client submit File code venv lib python3 10 site packages distributed client py line 1802 in submit futures = self graph to futures File code venv lib python3 10 site packages distributed client py line 2924 in graph to futures dsk = dsk dask distributed pack self keyset annotations File code venv lib python3 10 site packages dask highlevelgraph py line 1070 in dask distributed pack state layer dask distributed pack File code venv lib python3 10 site packages dask highlevelgraph py line 432 in dask distributed pack dsk = toolz valmap dumps task dsk File code venv lib python3 10 site packages toolz dicttoolz py line 85 in valmap rv update zip d keys map func d values File code venv lib python3 10 site packages distributed worker py line 2825 in dumps task d kwargs = warn dumps task 3 File code venv lib python3 10 site packages distributed worker py line 2837 in warn dumps b = dumps obj protocol=4 File code venv lib python3 10 site packages distributed protocol pickle py line 58 in dumps result = cloudpickle dumps x dump kwargs File code venv lib python3 10 site packages cloudpickle cloudpickle fast py line 73 in dumps cp dump obj File code venv lib python3 10 site packages cloudpickle cloudpickle fast py line 633 in dump return Pickler dump self obj pickle PicklingError Can t pickle lt function manifest files at 0x7f1eee165900 gt import of module prefect loader fetch manifests failed```

Oscar Björhn · Answer

I can t really help you other than to say it shouldn t be necessary Our flows run a bunch of tasks that are defined in other files

Anna Geller · Answer

it s a Dask issue

Anna Geller · Answer

on the prefect end there are no issues with putting tasks in a separate file but Dask cloudpickles everything to be able to send work to workers so this may not work with Dask even though it does work fine with just Prefect + e g ConcurrentTaskRunner

James Brady · Answer

Ah OK thanks Anna Obviously if we need to put everything in a single file that s not going to work super well grimacing I ll check with the Dask folks