Heya! So, we're eyeing how to transition from an Environment to Executors and Run_Configs. One blocker is that for some Flows, we change the cluster based on certain parameters. Have some ideas for how to implement that - any of these seem like they'd work well? 1. Flow-within-a-Flow that's registered with a different 

run_config

 based on params, and then kicked off from the Outer Flow 2. Spinning up a Dask Cluster from within the Flow and submitting tasks to it (and the Flow itself just runs in a small container) https://docs.prefect.io/core/idioms/resource-manager.html https://docs.prefect.io/api/latest/tasks/resources.html 3. Maybe start experimenting with Adaptive Scaling for Dask Clusters?

Hey Matt, the

DaskExecutor

cluster_class

kwarg can be an arbitrary callable so that's the first place I'd think of to adjust the cluster.

These are all viable approaches except I think with number 1, it’s not about the

RunConfig

but more about the

Executor

. You can use number 2 like this

Sweet, thanks!

Minimal example

import prefect as pf
from prefect.executors import DaskExecutor


def gen_cluster():
    from distributed import LocalCluster

    if pf.context.parameters["x"] < 10:

        return LocalCluster()

    else:

        return LocalCluster()


@pf.task
def my_task(x):
    pass


with pf.Flow("example", executor=DaskExecutor(cluster_class=gen_cluster)) as flow:
    x = pf.Parameter("x", 5)
    my_task(x)

flow.run()

Ooh, awesome!

I ran it with a breakpoint and could access the parameters locally. I guess I'm not 100% certain if they'll be available when run with an agent but I think they should be.

Cool, thanks! Will give it a go!

Let me know if they're not available from an agent -- I'd be willing to investigate a fix if so.

Awesome, thanks!

The RunConfig needs a matching Agent. Local Agent with LocalRun. ECS Agent with ECSRun. If you have multiple agents, maybe labels will help it get picked up by the right one

Aha, we've got `LocalAgent`s and `FargateAgent`s

FargateAgent might be deprecated? Not sure but ECSAgent is preferred

Yeah I'm seeing in the docs that the FargateAgent is deprecated