I have implemented distributed dask with the prefect. The task is working perfectly fine with LocalDaskExecutor but when I tried to run it in a Distributed Dask it has given an Error. I have a docker-compose of dask with 1 schdular and multiple workers. Every thing is up and works perfectly in case of local.

Hey @Atul Anand, could you move the code to the thread here? This issue seems like it’s because your Dask workers don’t have prefect installed. Did you install it on the cluster?

@task
def inc(x):
    sleep(random.random() / 10)
    return x + 1

@task
def dec(x):
    sleep(random.random() / 10)
    return x - 1

@task
def add(x, y):
    sleep(random.random() / 10)
    return x + y

@task(name="sum")
def list_sum(arr):
    return sum(arr)

with Flow("dask-example") as flow:
    incs = inc.map(x=range(100))
    decs = dec.map(x=range(100))
    adds = add.map(x=incs, y=decs)
    total = list_sum(adds)

if __name__ == "__main__":
    executor = DaskExecutor(
        debug=True,
        address="192.168.25.171:8786"
    )
    flow.executor = executor
    flow.run()

Have a look on this

That won’t have prefect but you can use the prefect image cuz it has dask too.

prefecthq/prefect

Ok , great. So I have one more question. Dask have schedular worker and notebook also. So all these things will be included in prefecthq/prefect image

Maybe not on the notebook. Yes the Prefect image can be used to spin up the scheduler and worker

Great @Kevin Kho and Thanks for Help. Let me try this once. Do you have any links or reference for docker compose file?

We dont have a docker compose file to spin up the cluster. More like the image just has the necessary dependencies. I think you can just edit the compose file you had earlier to change the image used?

@Kevin Kho Thanks a lot! Implemented that. Working like a charm🎯

NIce!