https://prefect.io logo
Join Slack
Channels
Powered by
Hello All! I am using prefect to run a Python scri...
# ask-community
s
Hello All! I am using prefect to run a Python script and I ran into an issue around a specific task that is querying a database and saving the results into the dataframe. Please see the error below. The query takes some tiem to execute ~40 seconds but is not particularly large only around 200MB. Any guidance would be helpful. Thanks! 
No heartbeat detected from the remote task; marking the run as failed.
k
Check this and there is a suggestion you can use to make this more stable
✅ 1
s
will check it out, thanks!
@Kevin Kho The parameter seems to be working on and off. When performing a quick run it executes without failure, however with a scheduled run it appears to be failing every now and again with the same error as above
k
How did you add the configuration? Seems like it didn’t stick for scheduled runs?
s
The parameter shows under the RUN CONFIG tab. I added it in the code directly under run_config.
k
Can I see a screen shot of that just to check it was set right?
s
the code or the tab or both?
k
Either
s
Copy code
run_config=DockerRun(
              image=IMAGE,
              labels=LABEL,
               env={"PREFECT__CLOUD__HEARTBEAT_MODE": "thread"}
          ),
 spirited-lorikeet OVERVIEWPARAMETERSRUN CONFIG
Copy code
{
  "env": {
    "PREFECT__CLOUD__HEARTBEAT_MODE": "thread"
  },
  "type": "DockerRun",
  "image": "spiderrockadvisors/py38-base:2.0.1",
  "ports": null,
  "labels": [
    "sradock01"
  ],
  "__version__": "1.2.0",
  "host_config": null
}
k
If this still fails, what we normally see is that the container is really dying/crashing. A good test is making this step into it’s own flow and turning off heartbeats and seeing if it runs to completion
s
Gotcha, will try that out. Is there a follow-up fix for if it fails within its own Flow?
k
Bump up memory or something maybe for the container it runs in
Most often it’s memory related
✅ 1
s
Thank you, I will look into it
@Kevin Kho So I have run multiple tests for this issue. The memory is not being overloaded max usage tops out at 180 mb, the timing is not running for very long and the Flow Run is only failing on scheduled runs still. Not sure what the next testing steps are. Still the same issue of 'No heartbeat detected from the remote task; marking the run as failed.'
k
Do you know how to reliably replicate? If you do we can definitely explore it more
s
Can you explain what that is?
k
Like do you have a code snippet we could look at and run to encounter the same issue?
Another thing you can do though is make that database querying step a subflow and turning off heartbeats for the subflow and then calling it with 
create_flow_run
s
Hmm, not sure that is viable given the data being used is sensitive information
k
Then I think you can either turn off heartbeats or make it a subflow without heartbeats
✅ 1
s
I will give it a go, thanks
5 Views
Open in Slack
PreviousNext
Powered by