Hi all! Yesterday during the last lab activity, I ...
# pacc-aug-28-29-2024
j
Hi all! Yesterday during the last lab activity, I and the group from Foundry had a question about how we can set flow-specific date variables that won't change on failure, and thus if we retry it in a few days, the date parameter will be the same. Hoping to automate this and standardize it across our ingestion flows. E.g., read/get variable from a location (perhaps it's set daily at 00:01 UTC), it's the "batch_cycle_date", and then use it for the current date. If a run fails, we'd like to retry the run (perhaps X days later), and the batch_cycle_date parameter is the same as when we first ran it. In my head, it naturally would be defined by the schedule (such that, on a schedule, one parameter is set and stays static). Then all of our runs would be idempotent w.r.t setting the date by nature. (We couldn't go back and run something from 3 days ago, and expect it to run for the current day - it would still run for the 3 days ago timestamp). We understand the persistence of parameters and overriding from the UI, but hoping to come to an automated resolution. Happy to elaborate further too! Perhaps an example solution could be found here? https://stackoverflow.com/questions/78483683/accessing-the-scheduled-time-from-within-a-prefect-flow-run
1
@Jeff Hale, @Kevin Grismore, @Bianca Hoch, tagging just in case. Thanks!
j
Just noting for posterity that I understand Kevin followed up in another channel. 🙂