Hui Zheng
12/11/2020, 1:41 AMprefecthq/prefect:0.13.15-python3.8
the flow run: https://cloud.prefect.io/semios/flow-run/19fcc78a-9442-48ef-8228-a9c3db18e341
Please see the threads for more details.Hui Zheng
12/11/2020, 1:44 AMDbtShellTask
, the timeout is set for 15 mins. In the past, when the task ran longer than that, a TimeoutError
exception were raised, which then we could handle according it.Hui Zheng
12/11/2020, 1:45 AMHui Zheng
12/11/2020, 2:11 AMdbtShellTask
run. by inspecting BigQuery logs, we could see that this task was running and then freezed after 10 mins of its execution.Hui Zheng
12/11/2020, 2:12 AMZanie
Hui Zheng
12/11/2020, 6:21 AMflow.environment = LocalEnvironment(labels=labels)
low.storage = Docker(...)
flow.register(... )
Zanie
Hui Zheng
12/11/2020, 5:01 PMlocalExecutor
and we don’t need LocalDaskExecutor
or DaskExecutor
because it’s pretty lightweight and won’t benefit from parallelism
.Hui Zheng
12/11/2020, 5:07 PMtimeouts are implemented in-process where your task is running
. Unfortunately our k8e job pod didn’t print out any more useful logs for this. Is there any logging you would suggest for troubleshooting this issue?Zanie
prefect.utilities.executors.run_with_thread_timeout
— it uses the prefect
logger with debug level logs so if you change set the PREFECT__LOGGING__LEVEL=DEBUG
in your execution environment you should be able to see the printouts