Samuel Kohlleffel
08/03/2021, 2:48 PMflow.serialized_hash()
to only register the modified flows. However, flow.serialized_hash()
is not returning a consistent hash value for flows that have not been modified. Why would this be the case?
For context, I'm testing by registering the flows locally with flow.register()
Kevin Kho
flow.register(…, idempotency_key=flow.serialized_hash())
, or are you just running flow.serialized_hash()
multiple times and comparing the values?Samuel Kohlleffel
08/03/2021, 2:56 PMflow.register(…, idempotency_key=flow.serialized_hash())
but that wasn't working for me, so then I tried to just test flow.serialized_hash()
by running it and comparing the values on independent registration runs.
I also compared the flow.serialize()
values on independent runs and those are different values as well.Kevin Kho
Samuel Kohlleffel
08/03/2021, 3:04 PMSamuel Kohlleffel
08/03/2021, 3:55 PMflow.serialize()
is the following:
Test 1: "start_date": {"dt": "2021-08-03T10:47:33.780910"
Test2: "start_date": {"dt": "2021-08-03T10:49:13.281844"
Does the time of registration impact the flow.serialize()
return value?Samuel Kohlleffel
08/03/2021, 4:01 PM"clocks": [{"__version__": "0.15.1", "cron": "20 6 * * *", "day_or": true, "end_date": null, "labels": null, "parameter_defaults": {}, "start_date": {"dt": "2021-08-03T10:49:13.281844", "tz": "America/Chicago"}, "type": "CronClock"}],
Kevin Kho
Samuel Kohlleffel
08/03/2021, 4:08 PMstart_date = pendulum.now(tz="America/Chicago")
clocks = [CronClock(s, start_date=start_date) for s in cron_strings]
run_schedule = create_cron_schedule("20 6 * * *")
with Flow(
name="netchb-to-snowflake",
schedule=netchb_run_schedule,
) as flow:
flow.chain(task chain...)
Samuel Kohlleffel
08/03/2021, 4:10 PMstart_date
parameter when building the Cron schedule. The start_date
value is then used in the flow.serialize()
json object which in turn causes inconsistent hash values because we are using pendulum.now(tz="America/Chicago")
as the start_date
.Kevin Kho
Kevin Kho
start_date
to the first of the year for something in the past so that the serialized_hash
will stay the same.Samuel Kohlleffel
08/03/2021, 5:11 PM