Kien Nguyen

    Kien Nguyen

    1 year ago
    Hi guys, has anyone have experience building the CI/CD pipeline for the flows, for example for every git push will re-register the flow? Cuz right now every times I push the code I will run register for all the flows again, even though the source code of the flows doesn't change
    Kevin Kho

    Kevin Kho

    1 year ago
    What Prefect version are you on and do you register with the CLI?
    Kien Nguyen

    Kien Nguyen

    1 year ago
    I'm calling the flow.register(). I think I have just found it. Does it solve the problem?
    flow.register(
        project_name="Hello, World!",
        idempotency_key=flow.serialized_hash()
    )
    Kevin Kho

    Kevin Kho

    1 year ago
    You can try using the CLI
    prefect register xxx
    . Also, I think the serialized hash might be changing if you have something like
    datetime.datetime.now()
    in your schedule?
    Can I see your schedule and Parameters?
    Kien Nguyen

    Kien Nguyen

    1 year ago
    with Flow('Historical Trades Yearly ETL', result=s3_result) as yearly_flow:
        yearly_dates = get_yearly_trading_dates()
        symbol = Parameter('symbol', 'SPY')
        limit = Parameter('limit', 10000)
        dates = Parameter('trading_dates', yearly_dates)
        loop_over_trading_dates(symbol, limit, dates)
    This one. Yeah so yearly_dates are dynamically generated.
    Kevin Kho

    Kevin Kho

    1 year ago
    That should be fine if it’s a task. How about schedule?
    Kien Nguyen

    Kien Nguyen

    1 year ago
    It is triggered via GQL/GUI
    So no schedule
    yearly_dates is not a task though
    Kevin Kho

    Kevin Kho

    1 year ago
    I see yeah it’s that causing it because that Parameter default is changing so when you re-register, there are changes. You can take a default like
    "current_year"
    and then have a task that provides the year if it received “current_year” as a string.
    Kien Nguyen

    Kien Nguyen

    1 year ago
    Hmm, if I convert
    get_yearly_trading_dates
    to a task and remove this parameter
    dates = Parameter('trading_dates', yearly_dates)
    it should solve the problem?
    Kevin Kho

    Kevin Kho

    1 year ago
    I think so