Pedro Machado
11/11/2021, 3:05 PMcreate_flow_run
task. The child flow takes over 3 hours and completed successfully. However, the parent flow's task failed with No heartbeat detected from the remote task; marking the run as failed.
I am using Prefect Cloud.
Please see thread for the code I am using to run the child flow. Thanks!Pedro Machado
11/11/2021, 3:06 PMKevin Kho
Pedro Machado
11/11/2021, 3:07 PMKevin Kho
wait_for_flow_run
?Pedro Machado
11/11/2021, 3:12 PMrun_ml_flow
is the 3h process that is dying.
ml_model_train_result = run_ml_flow(
parameters=flow_parameters(
script_path="projects/nte_optimization/train_main_remote_submit.py",
checkout_branch=ml_repo_branch,
),
name="train_main_remote_submit",
environment=flow_environment(iterations=nte_iterations),
)
ml_model_train_result.set_upstream(ml_model_prep_result)
ml_model_score_result = run_ml_flow(
parameters=flow_parameters(
script_path="projects/nte_optimization/scoring.py",
checkout_branch=ml_repo_branch,
),
name="scoring",
)
ml_model_score_result.set_upstream(ml_model_train_result)
Note that when the child flow takes less time to finish (for example a 20 min task) the parent flow works as expected. The issue seems to be related to the child flow taking a long time.Kevin Kho
Pedro Machado
11/11/2021, 3:14 PMPedro Machado
11/11/2021, 3:26 PMrun_ml_flow
? I an setting the run_name which helps identify the child flow runs, but both tasks show the same name run_ml_flow
on Prefect Cloud.Anna Geller
flow_run_id = create_flow_run(
flow_name="flow_to_start",
project_name="PROJECT_NAME",
task_args={"name": "your custom name"},
)
Kevin Kho
Pedro Machado
11/11/2021, 9:12 PMKevin Kho