Philip MacMenamin
05/20/2022, 9:42 PMKevin Kho
05/20/2022, 9:46 PMPhilip MacMenamin
05/20/2022, 9:48 PMKevin Kho
05/20/2022, 9:48 PMPhilip MacMenamin
05/20/2022, 9:49 PMKevin Kho
05/20/2022, 9:51 PMPhilip MacMenamin
05/20/2022, 9:51 PMKevin Kho
05/20/2022, 9:58 PMfrom image_portal_workflows.config import Config
hold? Anything Prefect related?Philip MacMenamin
05/20/2022, 10:00 PMKevin Kho
05/20/2022, 10:07 PMPhilip MacMenamin
05/20/2022, 10:13 PMKevin Kho
05/20/2022, 10:14 PMConfig
object that has two parts. One is the executor and number two are the directories. Whenever you pass Config
to a task, it passes both of these parts so it passes the Executor
that has the context
.
In order to send Config
to the Dask worker, it has to be serialized by cloudpickle (this is a Dask mechanism), so now Config
is getting pickled but then you have the context buried inside it which can’t be pickled.SLURM_exec
, but something doesn’t pass the “smell test” about having DaskExecutor
called in the config so I would just put SLURM_EXECUTOR = SLURM_exec
Philip MacMenamin
05/20/2022, 10:45 PM