What tends to cause this error for a task in the middle of a flow, when upstream tasks have run fine?

: No heartbeat detected from the remote task;

I typed up a FAQ answer here for that

@Kevin Kho Could you elaborate on this part of your answer? I tried configuring to use threads and still got the same error so now I'm onto the last solution:

In the event that the flow continues to fail after switching to using threads, some users have had success in making the failing task a subflow, and then turning off heartbeats for the subflow.

If you get the same error, I think it’s highly likely you are running out of memory. Also, how did you set the threaded heartbeats? For this one just one a new Flow with 1 task and then heartbeats off for that, and then call it with the

create_flow_run

task

I added this to my ECS run config at the bottom

Ah yeah that should work. I would suspect you have some memory issue as well and you can try bumping up the resources

Is there a reason I don't get the error when running it from my anaconda terminal but it does happen in production? This also happened when I tested out Prefect's case/when statement structure.

Do you have the same RAM on both?

I wouldn't know honestly, I'm just a user. The engineers set it up.

You can show me your ECS task definition or maybe you can get some insight running this with debug level logs?

I'll try the debug logs. What's the best way to do that?

PREFECT___LOGGING____LEVE_L:"DEBUG"

in the run config as part of the env. you can also show me the RunConfig and task definition if you want and I think we can increase the resources

Task: `

@task(name = "Transform", log_stdout=True, max_retries=3, retry_delay=timedelta(seconds=10))

Sorry task is overloaded lol. I mean ECS Task definition. I think the default is 2GB RAM

I don't think I ever specify the ECS task at any point

Ok let’s try bumping up that memory spec

How do I see the current memory allocated?

Maybe in the ECS UI?

kk, I'll look into it! Thank you.

There is a task definition and you can click into it and then you can scroll down

so I'd be adjusting the memory upward in the ECS run definition at the bottom of my flow?

Yes and that will override the default Prefect gives

Any general rule on how much to increase, if the default is 2084?

No general rule because I know you work with a bunch of dataframes so it depends on the data size