Hello can I get some help with debugging a failing workflow

Question

Philip MacMenamin · Answer

Kevin Kho · Answer

I can look at it but what am I suppose to keep an eye out for

Philip MacMenamin · Answer

hey Kevin thanks It s failing with a pickling error I m not sure why it s trying to pickle I guess

Kevin Kho · Answer

I can t guess from the logs Are you doing something like returning context

Kevin Kho · Answer

Or modifying it inside a task

Philip MacMenamin · Answer

I don t believe I m doing either of these things

Kevin Kho · Answer

Very hard to guess would you be able to DM me the code

Philip MacMenamin · Answer

it s very hard to understand what s going wrong with this logging system I m finding I feel like I maybe using it incorrectly

Philip MacMenamin · Answer

yes np

Philip MacMenamin · Answer

Philip MacMenamin · Answer

I ve updated the gist to reflect closer to what s going on Basically I m unable to understand the error logs so I incrementally uncomment stuff until I get an idea of what s failing

Kevin Kho · Answer

I can t see anything immediate What does `from image portal workflows config import Config` hold Anything Prefect related

Philip MacMenamin · Answer

Philip MacMenamin · Answer

I guess how do people normally debug workflows Because with larger flows I m finding it ~ impossible

Kevin Kho · Answer

Break it up into a flow of flows maybe

Kevin Kho · Answer

I am not super sure but I think it has to do with that the use of Prefect in that Config

Kevin Kho · Answer

Could you try a simple mapped task with a logging statement inside with that same setup

Philip MacMenamin · Answer

do you have an example of a flow of flows handy

Kevin Kho · Answer

Check <https docs prefect io core idioms flow to flow html scheduling a flow of flows|this>

Kevin Kho · Answer

Actually I know what happened I think You have a `Config` object that has two parts One is the executor and number two are the directories Whenever you pass `Config` to a task it passes both of these parts so it passes the `Executor` that has the `context` In order to send `Config` to the Dask worker it has to be serialized by cloudpickle this is a Dask mechanism so now `Config` is getting pickled but then you have the context buried inside it which can t be pickled

Kevin Kho · Answer

Try removing the logging from `SLURM exec` but something doesn t pass the smell test about having `DaskExecutor` called in the config so I would just put `SLURM EXECUTOR = SLURM exec`

Philip MacMenamin · Answer

ok I ll give these ideas a try Thanks Kevin