https://prefect.io logo
Title
b

Braun Reyes

01/07/2020, 1:14 AM
Hey there....wondering about '_requires_ result handlers for all flows with keyed edges that are registered with Prefect Cloud' in master branch. I rebased our fork and now some flows are failing to register. Is there a good way to understand this change? Is this so that flows runs can pick up from a last know successful task?
j

Jeremiah

01/07/2020, 1:58 AM
Hey @Braun Reyes - excellent question. We debated this a bit ourselves. As you know, Prefect Cloud can run a deferred execution of any task that 1) enters a retry state and 2) has an appropriate
ResultHandler
attached to serialize the data inputs. We believed for a long time that this was the “right” way to do it — data only gets serialized when it’s actually needed, so there are few performance/storage concerns. However, as the Cloud community has grown, we’ve discovered that this is actually not at all how most users perceive the system. Instead, it turns out there’s a very strong belief that any task can be rerun at any time — and worse, users only find out that isn’t true when it’s too late to go back and add the resulthandlers. This created a disappointing experience with failed runs (exactly the time we want Cloud to be most useful). Therefore, we decided to make “greedy serialization” the standard. For non-Docker environments, it was straightforward: we just added the new storage default as appropriate. And if we could have come up with a good default storage for Docker, we wouldn’t have put that error message in, but as you point out there isn’t a natural choice. We had some debate about how strongly to block it — in full disclosure, @Chris White believed it would be more disruptive than I did, so I take complete responsibility for your frustration! The upshot of this decision is that all users will be able to retry any task, no matter what state it’s in, and fully use the UI. Even though we still think greedy serialization isn’t always best, we’ve found it leads to a profoundly better user experience because it insures against all loss. Plus it is much easier than attempting in vain to educate all users about result handlers. So I hope that provides some clarity for this decision. I apologize for the inconvenience you encountered - we’ll see if there’s some way we could do this more gracefully for existing users, or provide a cleaner migration path.
c

Chris White

01/10/2020, 2:14 AM
Hey @Braun Reyes - based on this + some other user feedback, we decided to weaken the result handler requirement to a warning instead of an error. If you re-register any flows off of master, you won’t be required to pass a result handler. As always, thank you for the proactive feedback!
b

Braun Reyes

01/10/2020, 2:21 AM
Thank you...I am glad to help