Thread
#prefect-server
    Lawrence Finn

    Lawrence Finn

    1 year ago
    I am just starting to play with prefect so maybe I’m doing something dumb. I’m trying to use prefect with a fargate dask temporary cluster. The tasks run and finish, but the flow gets stuck in the running state even though each task finished successfully. It works fine with local executor, but not with dask fargate temporary cluster. Any ideas?
    Kevin Kho

    Kevin Kho

    1 year ago
    Hi @Lawrence Finn, I have this issue with DaskExectuor where the last hangs sometimes. Maybe you can try using a DaskExecutor with processes instead of threads.
    Lawrence Finn

    Lawrence Finn

    1 year ago
    how do i do that?
    Kevin Kho

    Kevin Kho

    1 year ago
    You would set it in the Dask config but one sec let me confirm with the team
    Could you show me how you set up the DaskExecutor?
    Lawrence Finn

    Lawrence Finn

    1 year ago
    certainly! and sorry if im doing something super dumb lol
    def fargate_cluster(n_workers=4):
        """Start a fargate cluster using the same image as the flow run"""
        return FargateCluster(
            image="actioniq/dask-image:latest",
            task_role_arn="...",
            execution_role_arn="...",
            n_workers=1,
            scheduler_cpu=256,
            scheduler_mem=512,
            worker_cpu=256,
            worker_mem=512,
            scheduler_timeout="15 minutes",
            subnets=["...],
            security_groups=["..."],
            skip_cleanup=True,
            fargate_use_private_ip=True,
        )
    executor = DaskExecutor(
        cluster_class=fargate_cluster,
        cluster_kwargs={
            "n_workers": 5,
        },
    )
    with Flow("docker-flow-3", storage=S3(bucket="....", key="meowsers/powsers"), schedule=schedule, executor=executor) as flow:
    OH i think it is bc of security groups, dask workers use some random port
    Kevin Kho

    Kevin Kho

    1 year ago
    Oh ok i’ll respond in a bit how to make the Fargate cluster use processes though, but that is worth a shot
    Lawrence Finn

    Lawrence Finn

    1 year ago
    yeah, the security group fixed what i was seeing… aws always gets ya 😄
    Kevin Kho

    Kevin Kho

    1 year ago
    Oh that’s good to know! How did you work around it? Open more ports?
    Lawrence Finn

    Lawrence Finn

    1 year ago
    yeah opening more ports worked. i think you can pass worker ports as a parameter, ill give that whirl too just so i dont need such permissive security groups (even though they are limited to 10.0.0.0/8)
    worker_extra_args=["--worker-port=8786"],