Hi all -- are there plans to add a 'Paused' state ...
# prefect-community
r
Hi all -- are there plans to add a 'Paused' state / signal / other mechanism to 2.0 for flows and/or tasks?
1
a
You can already pause deployments and work queues from the UI But regarding manual-only trigger and things like pause/resume flow run pattern, we will likely explore that in the future. I'm curious what's your use case for that feature? It might be possible to solve it in some other way already
r
Ah that's good to hear, and thanks for the quick reply! 🙂 There are a couple of use cases I have in mind, though right now I'm focusing on a case where we may need to retry a task after an extended waiting period e.g. 15h. For example: a service I'm using might impose a quota of 10000 requests per unique identifier per day. Each of my unique identifiers corresponds to a single device in my fleet of ocean wave height sensors, and I'm attempting to load several years' worth of historical data (structured as hourly dump files from another system) into the service as quickly as possible. Our wave sensors only intensively collect data when the waves reach a certain height within a given time period, so during any given week we may have very little data or maybe a whole lot of data from a given sensor. We read from the dump files and send batches of lines to the system, and every so often we will receive a 429/"ID quota exceeded for identifier xyz" error. If that happens, we should pause processing of that file for several hours to avoid spamming their system with requests they won't accept. Note that other files can continue to be processed in parallel, though if they happen to contain records from the same identifier then they'll get throttled as well (and should also be paused).
For this case ^ it would also be good if someone could manually un-pause the flow/task, for instance in the case that the service's wonderful customer support folks are willing to give us a quota exception for a few of our IDs.
a
would specifying retries only for specific exception types solve your issue? looks like this way you could retry only when it's needed
I believe the current functionality of pausing a deployment allows you to solve your use case already, even in a much simpler way than pause/resume states can - would you give it a try?