Or modifying it inside a task?

yes, np

I''ve updated the gist to reflect closer to what's going on. Basically I'm unable to understand the error logs, so I incrementally uncomment stuff until I get an idea of what's failing.

I guess how do people normally debug workflows? Because with larger flows I'm finding it ~ impossible

I am not super sure but I think it has to do with that the use of Prefect in that Config

Could you try a simple mapped task (with a logging statement inside) with that same setup?

Actually I know what happened I think. You have a

Config

object that has two parts. One is the executor and number two are the directories. Whenever you pass

Config

to a task, it passes both of these parts so it passes the

Executor

that has the

context

. In order to send

Config

to the Dask worker, it has to be serialized by cloudpickle (this is a Dask mechanism), so now

Config

is getting pickled but then you have the context buried inside it which can’t be pickled.

Try removing the logging from

SLURM_exec

, but something doesn’t pass the “smell test” about having

DaskExecutor

called in the config so I would just put

SLURM_EXECUTOR = SLURM_exec