from within tasks. I cannot guarantee this is recommended but it appears possible. Here is an example from another thread where someone seems to be doing this: https://prefect-community.slack.com/archives/CL09KU1K7/p1634664171072100?thread_ts=1634556225.467000&cid=CL09KU1K7 I will forward this question to see if I can get you a better answer
A is config B is creating a ML model in a very specific format that crashes the serialiser if I checkpoint it C is a computation that takes about 1½ hours but that may be interrupted D is a save-to-storage task that is happily retried and is idempotent How does all of this work together? I'm specifically looking at crash-resume remantics (not grey failure here). A crashes: rerun from start, B: rerun from cached A, C: rerun from cached A, then B, because B is not checkpointed, D : retry with cached result from C? Now if I add
A -> B(checkpoint=False) -> C(checkpoint=True) -> D
into the mix, am I right to assume that if the parent node, say (a) crashes, while work has been distributed across (b) and (c) also, the flow goes like the previous paragraph, but if (b) crashes, Dask will transparently rerun it?
error I think