Atul Anand

    Atul Anand

    5 months ago
    I have implemented distributed dask with the prefect. The task is working perfectly fine with LocalDaskExecutor but when I tried to run it in a Distributed Dask it has given an Error. I have a docker-compose of dask with 1 schdular and multiple workers. Every thing is up and works perfectly in case of local.
    Kevin Kho

    Kevin Kho

    5 months ago
    Hey @Atul Anand, could you move the code to the thread here? This issue seems like it’s because your Dask workers don’t have prefect installed. Did you install it on the cluster?
    Atul Anand

    Atul Anand

    5 months ago
    @task
    def inc(x):
        sleep(random.random() / 10)
        return x + 1
    
    @task
    def dec(x):
        sleep(random.random() / 10)
        return x - 1
    
    @task
    def add(x, y):
        sleep(random.random() / 10)
        return x + y
    
    @task(name="sum")
    def list_sum(arr):
        return sum(arr)
    
    with Flow("dask-example") as flow:
        incs = inc.map(x=range(100))
        decs = dec.map(x=range(100))
        adds = add.map(x=incs, y=decs)
        total = list_sum(adds)
    
    if __name__ == "__main__":
        executor = DaskExecutor(
            debug=True,
            address="192.168.25.171:8786"
        )
        flow.executor = executor
        flow.run()
    Have a look on this
    I have used ghcr.io/dask/dask:latest this image
    Kevin Kho

    Kevin Kho

    5 months ago
    That won’t have prefect but you can use the prefect image cuz it has dask too.
    prefecthq/prefect
    Atul Anand

    Atul Anand

    5 months ago
    Ok , great. So I have one more question. Dask have schedular worker and notebook also. So all these things will be included in prefecthq/prefect image
    Kevin Kho

    Kevin Kho

    5 months ago
    Maybe not on the notebook. Yes the Prefect image can be used to spin up the scheduler and worker
    Atul Anand

    Atul Anand

    5 months ago
    Great @Kevin Kho and Thanks for Help. Let me try this once. Do you have any links or reference for docker compose file?
    Kevin Kho

    Kevin Kho

    5 months ago
    We dont have a docker compose file to spin up the cluster. More like the image just has the necessary dependencies. I think you can just edit the compose file you had earlier to change the image used?
    Or you can add Prefect to the image you are using and make you rown
    Atul Anand

    Atul Anand

    5 months ago
    @Kevin Kho Thanks a lot! Implemented that. Working like a charm🎯
    Kevin Kho

    Kevin Kho

    5 months ago
    NIce!