Julian
09/15/2020, 11:51 AMfirst_of_the_week == first_of_the_month
? Is this possible, just using schedule and filters or do I need to
check this condition within the flow run and stop one of the schedule runs? It's crucial the flow run is not executed twice,
just because both conditions are metJim Crist-Harif
09/15/2020, 2:34 PMfilter
on a schedule. If you combine:
⢠A clock
to trigger an event every day
⢠A filter
to filter out only days that are the first of the week or the first of the month
you get your desired behavior. Something like:
import prefect
from prefect.schedules import clocks, Schedule
clock = clocks.CronClock("0 0 * * *")
def is_first_of_month_or_week(dt):
return dt.day == 1 or dt.weekday() == 1
schedule = Schedule(clocks=[clock], filters=[is_first_of_month_or_week])
Julian
09/15/2020, 4:38 PMTraceback (most recent call last):
File "stats/scale_cluster.py", line 121, in <module>
flow.register(project_name='default')
File "/usr/local/lib/python3.8/site-packages/prefect/core/flow.py", line 1588, in register
registered_flow = client.register(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 734, in register
serialized_flow = flow.serialize(build=build) # type: Any
File "/usr/local/lib/python3.8/site-packages/prefect/core/flow.py", line 1425, in serialize
serialized = schema(exclude=["storage"]).dump(flow_copy)
File "/usr/local/lib/python3.8/site-packages/marshmallow/schema.py", line 557, in dump
result = self._serialize(processed_obj, many=many)
File "/usr/local/lib/python3.8/site-packages/marshmallow/schema.py", line 521, in _serialize
value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
File "/usr/local/lib/python3.8/site-packages/marshmallow/fields.py", line 312, in serialize
return self._serialize(value, attr, obj, **kwargs)
File "/usr/local/lib/python3.8/site-packages/marshmallow/fields.py", line 567, in _serialize
return schema.dump(nested_obj, many=many)
File "/usr/local/lib/python3.8/site-packages/marshmallow_oneofschema/one_of_schema.py", line 72, in dump
result = result_data = self._dump(obj, **kwargs)
File "/usr/local/lib/python3.8/site-packages/marshmallow_oneofschema/one_of_schema.py", line 107, in _dump
result = schema.dump(obj, many=False, **kwargs)
File "/usr/local/lib/python3.8/site-packages/marshmallow/schema.py", line 557, in dump
result = self._serialize(processed_obj, many=many)
File "/usr/local/lib/python3.8/site-packages/marshmallow/schema.py", line 521, in _serialize
value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
File "/usr/local/lib/python3.8/site-packages/marshmallow/fields.py", line 312, in serialize
return self._serialize(value, attr, obj, **kwargs)
File "/usr/local/lib/python3.8/site-packages/marshmallow/fields.py", line 700, in _serialize
return [self.inner._serialize(each, attr, obj, **kwargs) for each in value]
File "/usr/local/lib/python3.8/site-packages/marshmallow/fields.py", line 700, in <listcomp>
return [self.inner._serialize(each, attr, obj, **kwargs) for each in value]
File "/usr/local/lib/python3.8/site-packages/prefect/utilities/serialization.py", line 397, in _serialize
raise ValidationError("Invalid function reference: {}".format(value))
marshmallow.exceptions.ValidationError: Invalid function reference: <function is_first_of_month_or_week at 0x7f0e290bd160>
Note that is_serializable
from prefect.utilities.debug
states that both the flow and the schedule are serializable.
Seems like in class NewScheduleSchema(ObjectSchema)
in prefect.serialization.schedule.py
there is a static list named FILTERS
of all allowed filters since
filters = fields.List(
StatefulFunctionReference(
valid_functions=FILTERS, reject_invalid=True, allow_none=True
)
)
rejects other filter-functions. Note that this also applies to or_filters,not_filters and adjustmentsnicholas
09/16/2020, 7:20 AMserialization.schedule.py
or changing reject_invalid
to False
. I don't think either are good solutions so let us look into this.Julian
09/16/2020, 7:22 AMnicholas
09/16/2020, 7:23 AMJulian
09/16/2020, 7:25 AMnicholas
09/16/2020, 5:07 PM_or_filters
here: https://github.com/PrefectHQ/prefect/pull/3330Julian
09/16/2020, 7:19 PMnicholas
09/16/2020, 7:21 PMis_day_of_week
will take a day in the range 0-6!Julian
09/16/2020, 8:04 PM