Dan Ball
07/02/2020, 5:54 PMDylan
07/02/2020, 7:36 PMSKIP
signal that might be useful for you in this instance. Since you want the flow to run every hour, the flow can parse the HTML and then reference the configuration timestamp. If the timestamps match, the Task can raise a SKIP
signal, and downstream tasks can set skip_on_upstream_skip=True
to skip as well. The Flow Run will be considered successful.
If the timestamps don’t match, the flow can proceed as normal. Here’s some documentation on the SKIP
signal:
https://docs.prefect.io/api/latest/engine/signals.html#skip
https://docs.prefect.io/core/concepts/states.html#skip-while-running
https://docs.prefect.io/api/latest/core/task.html#task-2Dan Ball
07/02/2020, 9:22 PMDylan
07/02/2020, 9:34 PM