Thread
#prefect-community
    Tom Klein

    Tom Klein

    2 months ago
    howdy 🙋 I have a couple of questions regarding best-practices with Prefect. • the first one is regarding a “validation” step (e.g. - if current state in the world already == desired state, then just SKIP the flow entirely). I know i can SKIP a task (by raising skip), but if i wanna skip the entire flow, should i just make sure that the downstream flows depend on it and that’s it? • the second Q is more general about orchestration : let’s say i have an external resource that i want to change to state “GREEN” at some time in the day (every day) and then set back to “BLUE” at some later time in the day (similarly, every day) - in the beginning i thought of doing like a single flow (that would run for example once an hour) and would contain an internal representation the desired schedule - and apply it within the run itself to decide if the state needs to be changed, and if so - to what. Then i figured it might be simple to just run the flow with two different parameters - twice a day (once to make things “Green”, and then again to change them back to “Blue”, with a flow-run parameter — is that even possible in prefect? to have two schedules with different parameters?) --- i’m not sure which one is deemed more “correct”?
    Anna Geller

    Anna Geller

    2 months ago
    #1 What would be the desired state? At a first glance, Terraform seems like a better tool for such use case #2 I'm not sure either
    Tom Klein

    Tom Klein

    2 months ago
    #1 the desired state changes based on the time of day. i.e. on 13:00 UTC we wanna change to green, and then on 22:00 UTC we wanna change to blue — every day, repeatedly. And this change is done via an API call (several actually). Why would Prefect not be suitable for that?
    Anna Geller

    Anna Geller

    2 months ago
    ahh thanks for more info I believe in that case, you should look more into caching - this way you can avoid recomputation for a given time
    Why would Prefect not be suitable for that?
    the description of the desired state led me to believe you may be looking into Infrastructure as Code
    Tom Klein

    Tom Klein

    2 months ago
    @Anna Geller the external resource is basically a cloud DB that we need to switch between "high" computation power and "low" computation power based on schedule (in the future we might change it to trigger based on the load on the DB itself). They don't have "auto-scaling" of resources so to avoid over-paying we need to scale it up and down ourselves. Not sure what you mean by recomputation? we don't actually compute anything, we just wanna alter the DB from one configuration to another back and forth (and e.g. keep it with the same "low" config throughout the weekend) at least for the first bullet i think i answered myself and basically if i raise
    SKIP
    in the validation task, it would cause all downstream tasks to skip too (and the flow itself, ultimately) - which achieves the goal i wanted, I'm just not sure how "correct" it is as far as best-practices go (e.g. would i have been better of with a conditional flow (i.e.
    case
    ) ? the problem with that is that i don't actually have anything to do if the condition is
    False
    for example, only if it's
    True
    ) for the second part -- i'm still unsure if having two "distinct" run-schedules (one for "green" and one for "blue") for the same flow make sense in an ideal world i would represent the entire schedule of what state should the DB be in at any given moment with like consecutive intervals that are represented in some compact way (and lives inside the flow itself) ---- rather than as just "cron-based schedules for change operations to run in" --- that way the flow could be run arbitrarily often and simply "adjust" the DB to the correct state when time fits the desired "mapping" (or, when other unique conditions apply -- perhaps) - if the DB doesn't happen to be in the desired state already kind of like polling behavior
    Anna Geller

    Anna Geller

    2 months ago
    seems like a very similar use case
    Tom Klein

    Tom Klein

    2 months ago
    hmm, not sure why is it similar? i don’t need to pause anything 🤷‍♂️
    Anna Geller

    Anna Geller

    2 months ago
    similar because it's also about DBA maintenance
    I honestly don't know how to help. Maintaining states of databases is not a problem that Prefect solves, Prefect is a tool to coordinate dataflows and work across teams, workflows and managing execution
    Tom Klein

    Tom Klein

    2 months ago
    @Anna Geller the fact that the resource happens to be a DB doesn't really have any bearing on the task (which is why I intentionally just called it an "external resource" at first and only elaborated later) - it could have been a "smart air-conditioner" that i'm trying to control the temperature for on schedule --- Prefect is just a convenient tool for running a bunch of tasks with scheduling, failure-handling, visibility, alerting, etc. the issue i'm asking about has to do with scheduling flows (and what's the best way to approach it), not with databases 🙂 also, it's a solved problem (my current solution work) - i'm more just consulting on what might be considered best-practice -- in the eyes of the orchestration experts and regarding polling - i think my use of the word might have been a bit incorrect --- i'm not concerned (yet) with reacting to an external state (i will in the future though) when i said "polling" i meant more like a high-frequency rule-enforcer rather than something that reacts to actual events in the real-world in other words, rather than Prefect flow running at hour A and hour B everyday based on two schedules (each with their own params "X" and "Y"), i could have a single flow running EVERY hour - and applying internally the logic that determines what should be the desired parameters at the current run - based on internally stored rules (e.g. if it's between 4AM and 5AM and today is a tuesday, do "X", otherwise, if it's between 9AM and 10M do "Y" and otherwise do nothing and leave things as they are") anyway, like i said, it's just kind of thinking out loud what would be "best" , it's not an actual unsolved problem - so not urgent either...
    Anna Geller

    Anna Geller

    2 months ago
    I don't have any best practice recommendations here, Prefect can cover a wide range of scheduling and you can e.g. even configure a custom RRULE schedule if things like interval or CRON don't work well for you glad this community helps you think out-loud and rubber duck 🙌 :marvin-duck: