https://prefect.io logo
Title
j

Jason Motley

03/07/2022, 4:59 PM
What tends to cause this error for a task in the middle of a flow, when upstream tasks have run fine?
: No heartbeat detected from the remote task;
k

Kevin Kho

03/07/2022, 5:01 PM
I typed up a FAQ answer here for that
❤️ 2
j

Jason Motley

03/08/2022, 3:28 PM
@Kevin Kho Could you elaborate on this part of your answer? I tried configuring to use threads and still got the same error so now I'm onto the last solution:
In the event that the flow continues to fail after switching to using threads, some users have had success in making the failing task a subflow, and then turning off heartbeats for the subflow.
k

Kevin Kho

03/08/2022, 3:30 PM
If you get the same error, I think it’s highly likely you are running out of memory. Also, how did you set the threaded heartbeats? For this one just one a new Flow with 1 task and then heartbeats off for that, and then call it with the
create_flow_run
task
j

Jason Motley

03/08/2022, 3:30 PM
I added this to my ECS run config at the bottom
env={"PREFECT__CLOUD__HEARTBEAT_MODE": "thread"}
k

Kevin Kho

03/08/2022, 3:33 PM
Ah yeah that should work. I would suspect you have some memory issue as well and you can try bumping up the resources
j

Jason Motley

03/08/2022, 3:34 PM
Is there a reason I don't get the error when running it from my anaconda terminal but it does happen in production? This also happened when I tested out Prefect's case/when statement structure.
k

Kevin Kho

03/08/2022, 3:35 PM
Do you have the same RAM on both?
j

Jason Motley

03/08/2022, 3:37 PM
I wouldn't know honestly, I'm just a user. The engineers set it up.
k

Kevin Kho

03/08/2022, 3:38 PM
You can show me your ECS task definition or maybe you can get some insight running this with debug level logs?
j

Jason Motley

03/08/2022, 3:39 PM
I'll try the debug logs. What's the best way to do that?
k

Kevin Kho

03/08/2022, 3:41 PM
PREFECT___LOGGING____LEVE_L:"DEBUG"
in the run config as part of the env. you can also show me the RunConfig and task definition if you want and I think we can increase the resources
Do you have CloudWatch logs though?
j

Jason Motley

03/08/2022, 3:42 PM
Task: `
@task(name = "Transform", log_stdout=True, max_retries=3, retry_delay=timedelta(seconds=10))
flow.run_config = ECSRun(
    labels=["ourlabel"],
    image="ourimage",
    env={"PREFECT__CLOUD__HEARTBEAT_MODE": "thread"})
Unsure re: cloud watch
k

Kevin Kho

03/08/2022, 3:43 PM
Sorry task is overloaded lol. I mean ECS Task definition. I think the default is 2GB RAM
j

Jason Motley

03/08/2022, 3:44 PM
I don't think I ever specify the ECS task at any point
k

Kevin Kho

03/08/2022, 3:49 PM
Ok let’s try bumping up that memory spec
Can you try filling cpu and memory? There is info in the docstring to what are acceptable values
j

Jason Motley

03/08/2022, 3:51 PM
How do I see the current memory allocated?
k

Kevin Kho

03/08/2022, 3:53 PM
Maybe in the ECS UI?
j

Jason Motley

03/08/2022, 3:54 PM
kk, I'll look into it! Thank you.
k

Kevin Kho

03/08/2022, 3:55 PM
There is a task definition and you can click into it and then you can scroll down
This is the default task definition so those are the default values your flow gets
j

Jason Motley

03/08/2022, 3:56 PM
so I'd be adjusting the memory upward in the ECS run definition at the bottom of my flow?
k

Kevin Kho

03/08/2022, 3:56 PM
Yes and that will override the default Prefect gives
j

Jason Motley

03/08/2022, 3:58 PM
Any general rule on how much to increase, if the default is 2084?
k

Kevin Kho

03/08/2022, 3:59 PM
No general rule because I know you work with a bunch of dataframes so it depends on the data size