Matic Lubej01/20/2021, 9:49 PM
or an orchestration backend for this?
Matic Lubej01/20/2021, 9:50 PM
Jim Crist-Harif01/20/2021, 9:51 PM
for your flow? My first guess is that results are being written on the worker nodes, so the results would be local to the worker filesystems not your flow-runner system. Likewise, logs are written to stdout on the nodes the tasks are run, so your task logs are logged on the workers not local to the
Matic Lubej01/20/2021, 9:54 PM
and what you are describing here makes a lot of sense.
Jim Crist-Harif01/20/2021, 9:57 PM
Matic Lubej01/20/2021, 10:01 PM
Jim Crist-Harif01/21/2021, 2:40 PM
configured to start a cluster on ECS. Or an ECS agent configured to use a local dask cluster (local meaning local to the same container as the flow runner).