<@ULVA73B9P> I have an ECS Prefect worker and subm...
# ask-marvin
d
@Marvin I have an ECS Prefect worker and submit tasks with a work pool. I want to use the Dask task runner so that it runs jobs in parallel. How would I do so and also be able to view the Dask dashboard that is created within the ECS task environment?
m
thought for 164.5 seconds
Short answer: use
DaskTaskRunner
inside your flow and make the Dask dashboard listen on 0.0.0.0. Then expose/allow that port (8787) on the ECS task so you can hit it from your machine. Yes, it’s all a bit more fun on Fargate. Here’s the pattern (Prefect 3.x) to run tasks in parallel in a single ECS task and enable the dashboard: 1) Make sure your flow image has Dask + prefect-dask installed - Add to your image:
Copy code
pip install "prefect[dask]" dask[distributed]
2) Use DaskTaskRunner in your flow - This starts a LocalCluster inside the ECS task (parallelism across the task’s CPUs). - Set the Dask dashboard to bind to all interfaces so ECS can expose it.
Copy code
from prefect import flow, task
from prefect_dask.task_runners import DaskTaskRunner

@task
def work(i):
    return i * i

@flow(
    task_runner=DaskTaskRunner(
        cluster_class="distributed.LocalCluster",
        cluster_kwargs={
            "n_workers": 4,                 # tune for your CPU
            "threads_per_worker": 1,        # CPU-bound: 1 thread each; I/O can use more
            "dashboard_address": "0.0.0.0:8787",
            # optional: "processes": True,
        },
    )
)
def my_flow():
    return [work.submit(i) for i in range(20)]

if __name__ == "__main__":
    my_flow()
3) Expose the dashboard from the ECS task - In your ECS task definition used by the work pool: - Add a container port mapping for 8787/tcp. - Use awsvpc networking and either: - Assign a public IP and open SG ingress to port 8787 from your IP, or - Put it behind an internal ALB (target group on port 8787) and access via ALB URL/VPN. - While the flow is running, find the task’s public IP (or ALB URL) in the ECS console and open http//&lt;that host&gt;8787. Notes and options - This setup gives you parallelism within a single ECS task (one machine). For multi-machine Dask, run a separate Dask scheduler/cluster and point
DaskTaskRunner(address="tcp://<scheduler>:8786")
at it; the dashboard would then be exposed by that cluster instead. - Security: only allow 8787 from your IP, and disable it when you’re done. The dashboard is not auth-protected. - Convenience: if you have a stable public URL (e.g., via ALB), you can publish it to the Prefect UI as an artifact:
Copy code
from prefect.artifacts import create_link_artifact
create_link_artifact(link="<http://your-alb-or-ip:8787>", description="Dask dashboard")
References - Prefect + Dask (prefect-dask) docs - Dask LocalCluster - Dask Dashboard docs - ECS port mappings If you share how your ECS work pool is set up (public vs private subnets, ALB or not), I can give you the exact network knobs to turn. I live to serve. Begrudgingly.