Hi again 😄 I just got the error

No heartbeat detected from the remote task; marking the run as failed

- this happend for tasks being run in parallel via the dask

KubeCluster

looking at our (internal) logs - it seems like all of them died simultaneously, 20+ minutes into their run there’s a total of 58 tasks being run, with 4 workers the first 4 tasks took approx. 1 hour to run and completed successfully, and then the next 4 that were run all seem to have failed simultaneously any idea / help?

We are aware and we currently don't have a solution for this in 1.0, we'll investigate better handling of that in 2.0 later this year this post discusses it more https://discourse.prefect.io/t/flow-is-failing-with-an-error-message-no-heartbeat-detected-from-the-remote-task/79

But is there a way to tell if the problem is at the flow or the task level?

task level

🤔 hmm.. ok i’ll try the

threads

thing. trying to restart the flow (with the restart button, i.e. with checkpointing) failed btw with the k8s error:

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"services \"tmpdask\" already exists","reason":"AlreadyExists","details":{"name":"tmpdask","kind":"services"},"code":409}

so i guess we should give our temporary dask cluster a temporary name, though i’m not sure it will help? isn’t a flow restart considered to have the same

flow_run_name

? but does the

threads

option affect how the FLOW sends the heartbeat, or how the tasks send heartbeats to the flow?

there are no flow level restarts in 1.0, but we have those in 2.0

wait, but i did do a flow restart just now? is that not considered a flow restart? i stood on the flow run and clicked

restart

at the top right doesn’t it run the flow again and use checkpointing to fetch results of successfully completed tasks…?

so sorry - I meant retries restarts work in 1.0 - you're 100% right

oh, that’s fine, i was talking about the fact that manually restarting the

flow

ended up failing because it tried to recreate the

dask cluster

with the same name (i guess?) - while the previous one was still a zombie --- which led to failure at least that’s what i understand from the error. so the question is - does restarting a flow lead to a different

flow run name

or

flow run id

or something that i can use (from

prefect.context

) to make sure that the dask cluster has a unique name? obviously if i use

flow_run_name

it would be unique per flow run, the question is - would it also be unique across flow restarts? would these arguments:

namespace="prefect", name=f"tmp_dask_{prefect.context.flow_run_name}", env=my_env

be sufficient to guarantee the cluster has a unique name across restarts? or do i need to use

flow_run_id

? or something else?

thanks for sharing your solution

@Anna Geller i ran into this during the execution (which is still ongoing):

Task 'wrapped_enrich_shell_task[55]': Finished task run for task with final state: 'Running'

does that make any sense? how can

Running

being a final state?