a

    Abhas P

    1 year ago
    Hi! I am trying to run a flow with mapped tasks with a DaskExecutor on my machine, I have a couple of queries :1. will the overall run time benefit if I run the same flow on Prefect Cloud? 2. Does prefect cloud manage the spawn of a Dask cluster resource ?
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Abhas P , local will be faster than Prefect cloud because each task needs to update its state with the Prefect API at least 3 times. These cause an overhead. Some DaskExecutors allow you to shut down after usage so it’s a setting when you do:
    executor = DaskExecutor(
        cluster_class="dask_cloudprovider.FargateCluster",
        cluster_kwargs={
            "image": "prefecthq/prefect:latest",
            "n_workers": 5,
            ...
        },
    )
    Otherwise you need to use the Prefect Resource Manager