Thread
#prefect-community
    ale

    ale

    1 year ago
    Hey folks 🙂 We have two agents, one used for testing and one for production. The testing agent is labelled "test", the production agent is labelled "production". We have a flow that we want to run on both environments, so we set both "test" and "production" labels on it. However, from Prefect docs I see that the agent labels must be a superset of the flow labels, otherwise the flow won't be picked up to run... What options do we have to run the same flow on two different agents? We are on Prefect 0.13.13, but it seems that this requirement has not changed in newer versions... Any suggestions are much appreciated!
    In addition, we would want to have different schedules for each environment. For example, when running with test agent we want a daily schedule, while when running with prod agent we want an hourly schedule
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @ale, if the flow has both labels, Prefect won’t know which one to send it to so here are some options. 1. Use clocks with a schedule. Each clock can take in different labels. 2. Manually trigger the flow and add the labels then. 3. Re-register under two different flows with different labels. You can do this in the same Python script.
    ale

    ale

    1 year ago
    Hey @Kevin Kho! The clock options seems to be the best options. Can this be done also from UI? Also the second one is very interesting. The third one is not viable for us. Would you mind providing an example for option 1 and 2?
    Kevin Kho

    Kevin Kho

    1 year ago
    It doesn’t seem like it’s exposed in the UI. Let me asked the UI team.
    Confirmed that this is not exposed in the UI, only in the programming interface. I’ll open a ticket for this.
    ale

    ale

    1 year ago
    Ok thanks! If you can provide even a simple example with clocks, it could be super useful by the way 🙂
    Also trigger the flow manually and providing a label is a very interesting options, because we wouldn't need to change the prod schedule which is hardcoded. Looking forward for an example of this as well 😅
    Kevin Kho

    Kevin Kho

    1 year ago
    The issue is here: https://github.com/PrefectHQ/ui/issues/962. The clocks can be found here:
    import datetime
    from prefect.schedules import clocks, Schedule
    
    now = datetime.datetime.utcnow()
    
    clock1   = clocks.IntervalClock(start_date=now, 
                                    interval=datetime.timedelta(minutes=1), 
                                    parameter_defaults={"p": "CLOCK 1"}, labels=["prod"])
    clock2   = clocks.IntervalClock(start_date=now + datetime.timedelta(seconds=30), 
                                    interval=datetime.timedelta(minutes=1), 
                                parameter_defaults={"p": "CLOCK 2"}, labels=["dev"])
    
    # the full schedule
    schedule = Schedule(clocks=[clock1, clock2])
    You would just click Run Now and change the labels here, but this is not with a schedule
    ale

    ale

    1 year ago
    Changing the label while running manually seems to be not available in 0.13.13, right?
    Kevin Kho

    Kevin Kho

    1 year ago
    Are you on Server or Cloud?
    ale

    ale

    1 year ago
    Server
    Kevin Kho

    Kevin Kho

    1 year ago
    I guess you may have to update then
    ale

    ale

    1 year ago
    Ok thanks @Kevin Kho 🙌
    a

    Alex Papanicolaou

    1 year ago
    Hey @ale, we kind of do something like this. We use a flow builder we developed to handle a lot of boilerplate stuff and for flows we want to be scoped like this, it automatically adjusts the name and the label to either staging or prod. We use the name to create separate versions of the flow that can run against different versions and we use the label to control which agent runs the different flows. When deploying the agents, we use an environment variable to signal to library code the context so it knows things like which databases or APIs to hit. By doing this with separate flows, it also means we can have staging flows progress and receive updates while prod is still run against a stable version of the docker image.
    ale

    ale

    1 year ago
    Interesting! Would be great to see an example 🙂
    a

    Alex Papanicolaou

    1 year ago
    I’ll see what @Marwan Sarieddine and I can do to elaborate. We have it in our backlog to describe our approaches