Ben Wenger

08/31/2023, 6:09 PM
Hey everyone! I'm trying to use the multiprocessing module in conjunction with prefect tasks/flows and I'm getting an error:
Copy code
Finished in state Failed("Task run encountered an exception: _pickle.PicklingError: Can't pickle <function ExtractLargeSQLServerData.extract_chunk_of_table_from_sql_server_to_blob at 0x00000209400FB600>: import of module '__prefect_loader__' failed\n")
extract_chunk_of_table_from_sql_server_to_blob is the function I'm trying to spin up in parallel. is there a way to do this with multiprocessing or do I need to rewrite it to use Prefect's built in dask parallel processing?