Matic Lubej
01/20/2021, 9:49 PMChris White
flow.run
or an orchestration backend for this?Matic Lubej
01/20/2021, 9:50 PMflow.run
, yes.Jim Crist-Harif
01/20/2021, 9:51 PMLocalResult
for your flow? My first guess is that results are being written on the worker nodes, so the results would be local to the worker filesystems not your flow-runner system.
Likewise, logs are written to stdout on the nodes the tasks are run, so your task logs are logged on the workers not local to the flow.run
call.Chris White
Matic Lubej
01/20/2021, 9:54 PMLocalResult
and what you are describing here makes a lot of sense.Chris White
Jim Crist-Harif
01/20/2021, 9:57 PMMatic Lubej
01/20/2021, 10:01 PMJim Crist-Harif
01/21/2021, 2:40 PMDaskExecutor
configured to start a cluster on ECS. Or an ECS agent configured to use a local dask cluster (local meaning local to the same container as the flow runner).