David Martin Calalang
08/19/2025, 5:37 PMMarvin
08/19/2025, 5:38 PMMarvin
08/19/2025, 5:40 PMDaskTaskRunner
inside your flow and make the Dask dashboard listen on 0.0.0.0. Then expose/allow that port (8787) on the ECS task so you can hit it from your machine. Yes, it’s all a bit more fun on Fargate.
Here’s the pattern (Prefect 3.x) to run tasks in parallel in a single ECS task and enable the dashboard:
1) Make sure your flow image has Dask + prefect-dask installed
- Add to your image:
pip install "prefect[dask]" dask[distributed]
2) Use DaskTaskRunner in your flow
- This starts a LocalCluster inside the ECS task (parallelism across the task’s CPUs).
- Set the Dask dashboard to bind to all interfaces so ECS can expose it.
from prefect import flow, task
from prefect_dask.task_runners import DaskTaskRunner
@task
def work(i):
return i * i
@flow(
task_runner=DaskTaskRunner(
cluster_class="distributed.LocalCluster",
cluster_kwargs={
"n_workers": 4, # tune for your CPU
"threads_per_worker": 1, # CPU-bound: 1 thread each; I/O can use more
"dashboard_address": "0.0.0.0:8787",
# optional: "processes": True,
},
)
)
def my_flow():
return [work.submit(i) for i in range(20)]
if __name__ == "__main__":
my_flow()
3) Expose the dashboard from the ECS task
- In your ECS task definition used by the work pool:
- Add a container port mapping for 8787/tcp.
- Use awsvpc networking and either:
- Assign a public IP and open SG ingress to port 8787 from your IP, or
- Put it behind an internal ALB (target group on port 8787) and access via ALB URL/VPN.
- While the flow is running, find the task’s public IP (or ALB URL) in the ECS console and open http//<that host>8787.
Notes and options
- This setup gives you parallelism within a single ECS task (one machine). For multi-machine Dask, run a separate Dask scheduler/cluster and point DaskTaskRunner(address="tcp://<scheduler>:8786")
at it; the dashboard would then be exposed by that cluster instead.
- Security: only allow 8787 from your IP, and disable it when you’re done. The dashboard is not auth-protected.
- Convenience: if you have a stable public URL (e.g., via ALB), you can publish it to the Prefect UI as an artifact:
from prefect.artifacts import create_link_artifact
create_link_artifact(link="<http://your-alb-or-ip:8787>", description="Dask dashboard")
References
- Prefect + Dask (prefect-dask) docs
- Dask LocalCluster
- Dask Dashboard docs
- ECS port mappings
If you share how your ECS work pool is set up (public vs private subnets, ALB or not), I can give you the exact network knobs to turn. I live to serve. Begrudgingly.