https://prefect.io logo
Title
h

Hannah Amundson

07/10/2020, 12:40 PM
Good morning! I have a flow that uses a self configured version of spark to write to s3. When I run this code in a kubernetes pod on my own without prefect code, it works fine. However, when I run this code using KubernetesJobEnvironment and prefect tasks/flow, it constantly says the heartbeat cannot be detected once it gets to the task that writes from one folder in s3 to another. Does anyone have any ideas on how to troubleshoot this? The problem appears to be with prefect or some interaction with prefect + spark/s3.
n

nicholas

07/10/2020, 1:22 PM
Hi @Hannah Amundson - I'm not sure why that would be the case for that take in particular unless Prefect is running into some memory issues. Are you using Prefect Cloud or Prefect Server?
h

Hannah Amundson

07/10/2020, 2:00 PM
@nicholas Prefect cloud
@nicholas how could i dig in to see if it is memory issues?
n

nicholas

07/10/2020, 6:02 PM
Ah sorry @Hannah Amundson I missed your response - one thing you can try to debug is to turn off heartbeats for the flow: you can do so by navigating to the Flow's page in the UI, going to the Settings tab and toggling off the Heartbeat setting
h

Hannah Amundson

07/10/2020, 8:46 PM
@nicholas Okay thank you! What does turning off the hearbeats do?
n

nicholas

07/10/2020, 8:47 PM
I think the docs will explain it better than me: https://docs.prefect.io/orchestration/concepts/flows.html#toggling-heartbeats 😄