https://prefect.io logo
a

Atul Anand

03/31/2022, 2:48 PM
I have implemented distributed dask with the prefect. The task is working perfectly fine with LocalDaskExecutor but when I tried to run it in a Distributed Dask it has given an Error. I have a docker-compose of dask with 1 schdular and multiple workers. Every thing is up and works perfectly in case of local.
k

Kevin Kho

03/31/2022, 2:51 PM
Hey @Atul Anand, could you move the code to the thread here? This issue seems like it’s because your Dask workers don’t have prefect installed. Did you install it on the cluster?
a

Atul Anand

03/31/2022, 2:53 PM
Copy code
@task
def inc(x):
    sleep(random.random() / 10)
    return x + 1

@task
def dec(x):
    sleep(random.random() / 10)
    return x - 1

@task
def add(x, y):
    sleep(random.random() / 10)
    return x + y

@task(name="sum")
def list_sum(arr):
    return sum(arr)

with Flow("dask-example") as flow:
    incs = inc.map(x=range(100))
    decs = dec.map(x=range(100))
    adds = add.map(x=incs, y=decs)
    total = list_sum(adds)

if __name__ == "__main__":
    executor = DaskExecutor(
        debug=True,
        address="192.168.25.171:8786"
    )
    flow.executor = executor
    flow.run()
Have a look on this
I have used ghcr.io/dask/dask:latest this image
k

Kevin Kho

03/31/2022, 2:56 PM
That won’t have prefect but you can use the prefect image cuz it has dask too.
prefecthq/prefect
🙌 1
a

Atul Anand

03/31/2022, 3:01 PM
Ok , great. So I have one more question. Dask have schedular worker and notebook also. So all these things will be included in prefecthq/prefect image
k

Kevin Kho

03/31/2022, 3:05 PM
Maybe not on the notebook. Yes the Prefect image can be used to spin up the scheduler and worker
👍 1
a

Atul Anand

03/31/2022, 3:07 PM
Great @Kevin Kho and Thanks for Help. Let me try this once. Do you have any links or reference for docker compose file?
k

Kevin Kho

03/31/2022, 3:08 PM
We dont have a docker compose file to spin up the cluster. More like the image just has the necessary dependencies. I think you can just edit the compose file you had earlier to change the image used?
Or you can add Prefect to the image you are using and make you rown
a

Atul Anand

03/31/2022, 4:19 PM
@Kevin Kho Thanks a lot! Implemented that. Working like a charm🎯
k

Kevin Kho

03/31/2022, 4:21 PM
NIce!
5 Views