https://prefect.io
Join Slack
We have a CI/CD pipeline that automatically registers flows for us (e.g. when a PR is opened); in my...
# ask-community
c
We have a CI/CD pipeline that automatically registers flows for us (e.g. when a PR is opened); in my registration script I do something like:
Copy code
if not args.dry_run:
        logger.log(SUCCESS, f"Registering flow {flow.name}")
        flow.register(project_name=args.project, idempotency_key=flow.serialized_hash())
Everything is working as expected, except I'm getting more versions of the flow (after multiple registrations) than I was previously expecting. It makes me wonder what I don't understand about the 
idempotency_key
- could someone explain to me what could cause the serialized_hash to change unexpectedly? for example, if a different VM registers flows, would the hash change?
k
Hey @Constantino Schillebeeckx, something very common is a default parameter with something like 
datetime.datetime.now()
could change the serialized hash. Do you have something like this?
c
Copy code
import pendulum
from decouple import config
from prefect import Flow
from prefect.schedules import CronSchedule

from flows.utils.dbt import DbtTask

dbt = DbtTask(profiles_dir=config("DBT_PROFILES_DIR"), return_all=True, log_stderr=True)

with Flow("dbt_test") as flow:

    # run at 08:00 in whatever is the local timezone of the script
    flow.schedule = CronSchedule("0 8 * * *", start_date=pendulum.now())

    dbt(command="dbt test")


if __name__ == "__main__":
    flow.run()  # so we can easily test
is one example, I'm guessing that 
start_date
is the guilty party?
k
Yeah that’s the guilty party. Just set it to the start of this year.
c
🏆 awesome - thank you so much for always being available 😄
👍 1
k
Wait, I think it’s more appropriate to set the schedule outside the 
with
clause btw, though I think this will work
c
you mean like:
Copy code
schedule = CronSchedule(...)

with FLow('name', schedule=schedule) as flow:
k
Copy code
with Flow(..) as flow:
    ....

flow.schedule = ...
c
in your example, the schedule won't get serialized?
k
It will. RunConfig and Storage too.
c
in that case, does it matter how I'm setting the scedule?
k
Uhh probably not, but outside is more common
🙌 1
4 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by