Saurabh Indoria
01/27/2022, 5:01 AMKevin Kho
Saurabh Indoria
01/27/2022, 5:42 AMKevin Kho
Saurabh Indoria
01/27/2022, 5:44 AMKevin Kho
Saurabh Indoria
01/27/2022, 5:55 AMKevin Kho
ClientError
or manual_only
trigger fit what you are doing? Or just the set_upstream
?Saurabh Indoria
01/27/2022, 6:04 AMKevin Kho
Saurabh Indoria
01/27/2022, 6:07 AMKevin Kho
processes
instead of threads
for LocalDaskExecutor?Saurabh Indoria
01/27/2022, 6:08 AMKevin Kho
LocalDaskExecutor(scheduler="processes")
and it might be more stable than threads. Yes setting from the UI will affect the Flow since the state is resolved already. So when a downstream checks the upstream dependencies, it will affect whether it starts or not.Saurabh Indoria
01/27/2022, 6:12 AM.map
so there are multiple parallel task runs. Some of these task runs would get stuck, like no useful logs other than task started running...
Does this sound like a LocalDaskExecutor issue?Kevin Kho
@task(timeout=timedelta(hours=1))
Saurabh Indoria
01/27/2022, 6:18 AMKevin Kho