https://prefect.io logo
d

David Cupp

10/11/2022, 9:44 PM
Does anyone know how to schedule something in prefect based on an rruleset? According to the docs [1] and source code [2] the
RRuleSchedule
only takes a "rrule string". It seems easy to convert a single rrule to a string, but as far as I can tell there is no standard implementation to convert an rrule set into an rrule string [3]. Any ideas? [1] https://docs.prefect.io/api-ref/orion/schemas/schedules/#prefect.orion.schemas.schedules.RRuleSchedule [2] https://github.com/PrefectHQ/prefect/blob/main/src/prefect/orion/schemas/schedules.py#L322 [3] https://github.com/dateutil/dateutil/issues/856
1
m

Mason Menges

10/11/2022, 9:50 PM
Hey @David Cupp I haven't tested out the RRuleSchedules too much myself yet but these are some example strings from their docs https://dateutil.readthedocs.io/en/stable/rrule.html#rrulestr-examples
d

David Cupp

10/11/2022, 9:52 PM
Sure, but
rrulestr()
converts from an "rrule string" to an
rruleset
(as necessary). I need the reverse of that operation. As far as I can tell, there is nothing here that can convert from the rruleset python object to an rrule string.
The closest I've come is seeing people who write their own rrule string function that looks at the private members of rruleset and converts them one by one, which might be an option if I'm not the only one doing it with prefect....
d

David Cupp

10/11/2022, 10:51 PM
Thanks, but those are examples for people who already have an rrule string. I have an "rrule set" which is a python object in the
dateutil
library: https://dateutil.readthedocs.io/en/stable/rrule.html#rruleset-examples I'm confident I know what do to once I have an rrule string, but getting to the string from the rrule set (without writing code that is too hacky) is the challenge.
a

Anna Geller

10/11/2022, 11:26 PM
@David Cupp what problem are you trying to solve on a high level? can you explain what kind of schedule would you like to attach to your deployment?
I'm interested in hearing something like: "every 2 days at 5pm" or "every 10 days", "daily for the next 7 days"
fwiw I totally agree that we need much more rule examples to clarify how to apply it tp various use cases
d

David Cupp

10/11/2022, 11:39 PM
At a very high level, I'm trying to integrate our proprietary system (or prorprietary representations of a schedule) with Prefect (or prefects notion of a schedule). We have multiple types of schedule definitions. The one that inspired my question here today is the one that our customers can edit in our UI, and which is stored in a database. Each of these schedules is implemented as a set of
ProprietaryConfig
objects. So this is not a case where I personally have a schedule in mind, and need help manually writing an rrule string for it. This is about writing a robust, stable conversion. We already have code that maps from
List[ProprietaryConfig]
-->
dateutil.rruleset
(our existing scheduler uses the rruleset object directly) so Prefect's
RRuleSchedule
seems like the best fit. But
RRuleSchedule
takes a "rrule string", so as far as I know my options are (from best to worst): 1. find some code that can convert
dateutil.rruleset
--> "rrule string" 2. find some code that can convert
List[dateutil.rrule]
--> "rrule string" 3. implement
List[PrioprietaryConfig]
--> "rrule string" 4. find out that we can use multiple
RRuleSchedule
objects or something else that I'm missing. Any of these solutions is possible, but of course I would prefer to avoid doing anything that involves relying on private fields of a python class, or creating our own rrule implmentation.
Does that answer your question? Also, fun fact -- I think our CTO is responsible for one of the early RRule PRs: https://github.com/PrefectHQ/prefect/pull/4901
a

Anna Geller

10/12/2022, 12:10 AM
interesting, perhaps you could start by scheduling your flows directly from your system? would that be an option?
d

David Cupp

10/12/2022, 12:12 AM
Sure, that is the likely migration plan (migrate DAG execution first). Eventually, I plan to have Prefect Cloud own the scheduler portion.
👍 1
a

Anna Geller

10/12/2022, 12:12 AM
it doesn't fully answer my question because I would like to find out why rruleset would be necessary. E.g., could you give me examples of schedules that you cannot generate using the current RRuleSchedule object and that would require us to support rruleset?
we would be open to adding that as a feature to the backlog if there is a clear motivating use case for it e.g. I can't schedule my flow to run at XXX because neither of the available options i.e. interval + cron + rrule string supports that pattern/use case
d

David Cupp

10/12/2022, 12:15 AM
could you give me examples of schedules that you cannot generate using the current RRuleSchedule object
I'm not trying to claim that Prefect is in any way inadequate, and I think that by supporting an rrule string it technically can handle any type of schedule that we have. My problem is simply an impedance mismatch with the data types (one that I blame python's
dateutil
library for). I'm feeling around for answers because, just like you don't want to roll your own crypto, I don't want to roll my own calendar logic/RFC implementations (unless I have to).
🙏 1
a

Anna Geller

10/12/2022, 12:16 AM
yeah I understand that, but this doesn't motivate a product feature yet, but more e.g. some utility that could help you make that conversion. Have you seen this workaround?
it's from this issue (there are more people who struggle with the same conversion as you do)
d

David Cupp

10/12/2022, 12:19 AM
Yeah that's the issue I linked to in my original post here. That is certainly a possibility. The obvious downside is that the moment we rely on an undocumented private member field, we are going to have to be very careful about upgrades to
dateutil
since each one could break our implementation.
I'm not locked into a
dateutil.rruleset
but I am sort of locked into a list of rule objects
a

Anna Geller

10/12/2022, 12:23 AM
Gotcha. If you would be open to contributing something like RRuleSetSchedule, we would certainly review your PR -- might be easier than I think. We just don't have the bandwidth atm to implement this, but this shouldn't stop you from opening a PR -- doesn't hurt for sure
d

David Cupp

10/12/2022, 12:26 AM
Thanks I will keep that in mind.
👍 2
7 Views