Jovan Visnjic

09/28/2020, 10:12 AM
Hi everyone. I have a question about
and exceptions. We use
for checkpointing, so only failed tasks get executed on a workflow re-run. So far we execute the workflow without a server, using just the basic CLI. If
exists, the task is skipped and just tasks with missing results get executed. We would also like to store error information on the disk. But if write exception's content as a
through task's
parameter, it gets treated as successful, because something exists at the result location, whatever that may be. Is there some established way of dumping errors to disk, so they don't just remain in the console's
, but that at the same time they don't obstruct the logic of what is successful and what not? Any ideas or comments appreciated. Jovan

Kyle Moon-Wright

09/28/2020, 3:12 PM
Hey @Jovan Visnjic, This is an interesting problem, and I think any solution will have pros and cons. Ultimately, any task errors will bubble to the flow, so you can cache at the flow level to catch any upstream task errors. Of course, there's no way to cache errors for tasks that don't run (due to a result existing), so you may need to write this in a way that your tasks always run, but have custom logging/caching depending on the existence of the result: to cache the result or cache an error. Also, I'm sure there's some silver bullet solution I'm not thinking of...

Jovan Visnjic

09/29/2020, 2:58 PM
Thanks @Kyle Moon-Wright. I am thinking of some custom result class, which would evaluate not just based on whether result exists, but also it's content. Does this make sense? This also touches a bit the other question of mine that you answered . Since we still have this problem to overcome, this custom result class could also handle both local and server cases. But this idea seems like "One master
class to rule them all", so I'm also skeptical about this approach... 🤔