Good morning! I have a flow that uses a self configured version of spark to write to s3. When I run this code in a kubernetes pod on my own without prefect code, it works fine. However, when I run this code using KubernetesJobEnvironment and prefect tasks/flow, it constantly says the heartbeat cannot be detected once it gets to the task that writes from one folder in s3 to another. Does anyone have any ideas on how to troubleshoot this? The problem appears to be with prefect or some interaction with prefect + spark/s3.

Hi @Hannah Amundson - I'm not sure why that would be the case for that take in particular unless Prefect is running into some memory issues. Are you using Prefect Cloud or Prefect Server?

@nicholas Prefect cloud

Ah sorry @Hannah Amundson I missed your response - one thing you can try to debug is to turn off heartbeats for the flow: you can do so by navigating to the Flow's page in the UI, going to the Settings tab and toggling off the Heartbeat setting

@nicholas Okay thank you! What does turning off the hearbeats do?

I think the docs will explain it better than me: https://docs.prefect.io/orchestration/concepts/flows.html#toggling-heartbeats 😄