https://prefect.io
Join Slack
What tends to cause this error for a task in the middle of a flow, when upstream tasks have run fine...
# ask-community
j
What tends to cause this error for a task in the middle of a flow, when upstream tasks have run fine? 
: No heartbeat detected from the remote task;
k
I typed up a FAQ answer here for that
❤️ 2
j
@Kevin Kho Could you elaborate on this part of your answer? I tried configuring to use threads and still got the same error so now I'm onto the last solution: 
In the event that the flow continues to fail after switching to using threads, some users have had success in making the failing task a subflow, and then turning off heartbeats for the subflow.
k
If you get the same error, I think it’s highly likely you are running out of memory. Also, how did you set the threaded heartbeats? For this one just one a new Flow with 1 task and then heartbeats off for that, and then call it with the 
create_flow_run
task
j
I added this to my ECS run config at the bottom
Copy code
env={"PREFECT__CLOUD__HEARTBEAT_MODE": "thread"}
k
Ah yeah that should work. I would suspect you have some memory issue as well and you can try bumping up the resources
j
Is there a reason I don't get the error when running it from my anaconda terminal but it does happen in production? This also happened when I tested out Prefect's case/when statement structure.
k
Do you have the same RAM on both?
j
I wouldn't know honestly, I'm just a user. The engineers set it up.
k
You can show me your ECS task definition or maybe you can get some insight running this with debug level logs?
j
I'll try the debug logs. What's the best way to do that?
k
PREFECT___LOGGING____LEVE_L:"DEBUG"
in the run config as part of the env. you can also show me the RunConfig and task definition if you want and I think we can increase the resources
Do you have CloudWatch logs though?
j
Task: `
Copy code
@task(name = "Transform", log_stdout=True, max_retries=3, retry_delay=timedelta(seconds=10))
Copy code
flow.run_config = ECSRun(
    labels=["ourlabel"],
    image="ourimage",
    env={"PREFECT__CLOUD__HEARTBEAT_MODE": "thread"})
Unsure re: cloud watch
k
Sorry task is overloaded lol. I mean ECS Task definition. I think the default is 2GB RAM
j
I don't think I ever specify the ECS task at any point
k
Ok let’s try bumping up that memory spec
Can you try filling cpu and memory? There is info in the docstring to what are acceptable values
j
How do I see the current memory allocated?
k
Maybe in the ECS UI?
j
kk, I'll look into it! Thank you.
k
There is a task definition and you can click into it and then you can scroll down
This is the default task definition so those are the default values your flow gets
j
so I'd be adjusting the memory upward in the ECS run definition at the bottom of my flow?
k
Yes and that will override the default Prefect gives
j
Any general rule on how much to increase, if the default is 2084?
k
No general rule because I know you work with a bunch of dataframes so it depends on the data size
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by