https://prefect.io
Join Slack
Hello - can I get some help with debugging a failing workflow?
# ask-community
p
Hello - can I get some help with debugging a failing workflow?
k
I can look at it, but what am I suppose to keep an eye out for?
p
hey Kevin, thanks. It's failing with a pickling error, I'm not sure why it's trying to pickle I guess
k
I can’t guess from the logs. Are you doing something like returning context?
Or modifying it inside a task?
p
I don't believe I'm doing either of these things
k
Very hard to guess, would you be able to DM me the code?
p
it's very hard to understand what's going wrong with this logging system I'm finding, I feel like I maybe using it incorrectly
yes, np
I''ve updated the gist to reflect closer to what's going on. Basically I'm unable to understand the error logs, so I incrementally uncomment stuff until I get an idea of what's failing.
k
I can’t see anything immediate. What does 
from image_portal_workflows.config import Config
hold? Anything Prefect related?
p
I guess how do people normally debug workflows? Because with larger flows I'm finding it ~ impossible
k
Break it up into a flow of flows maybe?
I am not super sure but I think it has to do with that the use of Prefect in that Config
Could you try a simple mapped task (with a logging statement inside) with that same setup?
p
do you have an example of a flow of flows handy?
k
Check this
Actually I know what happened I think. You have a 
Config
object that has two parts. One is the executor and number two are the directories. Whenever you pass 
Config
to a task, it passes both of these parts so it passes the 
Executor
that has the 
context
. In order to send 
Config
to the Dask worker, it has to be serialized by cloudpickle (this is a Dask mechanism), so now 
Config
is getting pickled but then you have the context buried inside it which can’t be pickled.
Try removing the logging from 
SLURM_exec
, but something doesn’t pass the “smell test” about having 
DaskExecutor
called in the config so I would just put 
SLURM_EXECUTOR = SLURM_exec
p
ok, I'll give these ideas a try. Thanks Kevin.
4 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by