https://prefect.io
Join Slack
Hi everyone. I am running into an issue where a task is failing due to heartbeat. I have a master fl...
# ask-community
p
Hi everyone. I am running into an issue where a task is failing due to heartbeat. I have a master flow that kicks off another flow with the 
create_flow_run
task. The child flow takes over 3 hours and completed successfully. However, the parent flow's task failed with 
No heartbeat detected from the remote task; marking the run as failed.
I am using Prefect Cloud. Please see thread for the code I am using to run the child flow. Thanks!
Untitled.py
Untitled.py
k
Are you waiting for the child flow run to finish?
p
Yes. There is another task that depends on this one.
k
So you use the 
wait_for_flow_run
?
p
No. Here is what I have now. The first call to 
run_ml_flow
is the 3h process that is dying.
Copy code
ml_model_train_result = run_ml_flow(
        parameters=flow_parameters(
            script_path="projects/nte_optimization/train_main_remote_submit.py",
            checkout_branch=ml_repo_branch,
        ),
        name="train_main_remote_submit",
        environment=flow_environment(iterations=nte_iterations),
    )
    ml_model_train_result.set_upstream(ml_model_prep_result)

    ml_model_score_result = run_ml_flow(
        parameters=flow_parameters(
            script_path="projects/nte_optimization/scoring.py",
            checkout_branch=ml_repo_branch,
        ),
        name="scoring",
    )
    ml_model_score_result.set_upstream(ml_model_train_result)
Note that when the child flow takes less time to finish (for example a 20 min task) the parent flow works as expected. The issue seems to be related to the child flow taking a long time.
k
Can you try doing this . Set the heartbeat to “thread” instead of the default “process”
p
Ok. I'll give it a try and report back in a few hours 🙂 Thanks Kevin.
👍 1
Kevin, in the meantime, how can I change the name of the tasks above if both use 
run_ml_flow
? I an setting the run_name which helps identify the child flow runs, but both tasks show the same name 
run_ml_flow
on Prefect Cloud.
a
Copy code
flow_run_id = create_flow_run(
        flow_name="flow_to_start",
        project_name="PROJECT_NAME",
        task_args={"name": "your custom name"},
    )
upvote 1
k
No rush, but really wanna hear if this worked @Pedro Machado
p
Hi Kevin. I completed one run. It 5 hours. It did not error. I will run it nightly for a few days to see if it happens. Thanks for your help!
k
Nice! Thanks for the info
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by