Hi everyone, question regarding putting flow runs to different agents, Prefect 1 (Prefect 2 is in future plans 😞 ) - see thread

I also have the same problem, but more for heterogeneous hardware

For prefect1 use labels!

I think if I am understanding correctly, its less of an issue of “this flow goes to this agent” and more of an issue of I have 100 flow runs, and I want to equally spread them out across multiple agents. Having to manually map this out with labels might be tricky, you’d need to implement some kind of check when a new flow run is added where you inspect your pool of agents, see which one is under utilized and set the label on the incoming flow run to match the agent

There is unfortunately no concept of distribution in Prefect1

ok, for those who are facing it seems I’ve found less painfull approach: we can pass non-default labels to

create_flow_run

so, it allows to implement some form of round-robin, I’ve ended up with assigning additional unique labels for each agent, and when creating flow run - just randomly pick one of it and add to default labels list, something like:

@task()
def process_flow_batch():
    
    default_labels = ["ec2", ... ]

    agent_specific_labels = ["agent1", ... ]

    flow_update_batch = create_flow_run.run(
        flow_name="some_flow_name",
        project_name="some_project_name",
        labels=default_labels.append(agent_specific_labels[random.randint(0, len(agent_specific_labels) - 1)]
    )

    wait_for_flow_run.run(
        flow_update_batch
    )