m

    Matt Alhonte

    11 months ago
    Heya! So, we're eyeing how to transition from an Environment to Executors and Run_Configs. One blocker is that for some Flows, we change the cluster based on certain parameters. Have some ideas for how to implement that - any of these seem like they'd work well?1. Flow-within-a-Flow that's registered with a different 
    run_config
     based on params, and then kicked off from the Outer Flow 2. Spinning up a Dask Cluster from within the Flow and submitting tasks to it (and the Flow itself just runs in a small container) https://docs.prefect.io/core/idioms/resource-manager.html https://docs.prefect.io/api/latest/tasks/resources.html 3. Maybe start experimenting with Adaptive Scaling for Dask Clusters?
    Michael Adkins

    Michael Adkins

    11 months ago
    Hey Matt, the
    DaskExecutor
    cluster_class
    kwarg can be an arbitrary callable so that's the first place I'd think of to adjust the cluster.
    Kevin Kho

    Kevin Kho

    11 months ago
    These are all viable approaches except I think with number 1, it’s not about the
    RunConfig
    but more about the
    Executor
    . You can use number 2 like this
    m

    Matt Alhonte

    11 months ago
    Sweet, thanks!
    Michael Adkins

    Michael Adkins

    11 months ago
    Minimal example
    import prefect as pf
    from prefect.executors import DaskExecutor
    
    
    def gen_cluster():
        from distributed import LocalCluster
    
        if pf.context.parameters["x"] < 10:
    
            return LocalCluster()
    
        else:
    
            return LocalCluster()
    
    
    @pf.task
    def my_task(x):
        pass
    
    
    with pf.Flow("example", executor=DaskExecutor(cluster_class=gen_cluster)) as flow:
        x = pf.Parameter("x", 5)
        my_task(x)
    
    flow.run()
    m

    Matt Alhonte

    11 months ago
    Ooh, awesome!
    Sounds like that'd also be the "Prefect-onic" way to do it?
    (#1 and the Higher-Order Flow kinda appealed to me but I think that's maybe flying a little too close to the sun lol)
    @Michael Adkins Are you sure it'd be able to access Parameters? This thread made it seem like it wouldn't https://prefect-community.slack.com/archives/CL09KU1K7/p1631644543279100
    Michael Adkins

    Michael Adkins

    11 months ago
    I ran it with a breakpoint and could access the parameters locally. I guess I'm not 100% certain if they'll be available when run with an agent but I think they should be.
    m

    Matt Alhonte

    11 months ago
    Cool, thanks! Will give it a go!
    Definitely seems the most elegant
    Michael Adkins

    Michael Adkins

    11 months ago
    Let me know if they're not available from an agent -- I'd be willing to investigate a fix if so.
    m

    Matt Alhonte

    11 months ago
    Awesome, thanks!
    Hrm, so tried to do a little Hello World for Run_Config + Executor, but got this: has a run_config of type ECSRun, only LocalRun is supported
    Kevin Kho

    Kevin Kho

    11 months ago
    The RunConfig needs a matching Agent. Local Agent with LocalRun. ECS Agent with ECSRun. If you have multiple agents, maybe labels will help it get picked up by the right one
    m

    Matt Alhonte

    11 months ago
    Aha, we've got LocalAgents and FargateAgents
    Kevin Kho

    Kevin Kho

    11 months ago
    FargateAgent might be deprecated? Not sure but ECSAgent is preferred
    m

    Matt Alhonte

    11 months ago
    Yeah I'm seeing in the docs that the FargateAgent is deprecated