https://prefect.io logo
Powered by
Title
s

Sushma Adari

06/01/2022, 6:39 PM
Hello All! I am using prefect to run a Python script and I ran into an issue around a specific task that is querying a database and saving the results into the dataframe. Please see the error below. The query takes some tiem to execute ~40 seconds but is not particularly large only around 200MB. Any guidance would be helpful. Thanks! 
No heartbeat detected from the remote task; marking the run as failed.
k

Kevin Kho

06/01/2022, 6:42 PM
Check this and there is a suggestion you can use to make this more stable
✅ 1
s

Sushma Adari

06/01/2022, 6:44 PM
will check it out, thanks!
@Kevin Kho The parameter seems to be working on and off. When performing a quick run it executes without failure, however with a scheduled run it appears to be failing every now and again with the same error as above
k

Kevin Kho

06/02/2022, 2:32 PM
How did you add the configuration? Seems like it didn’t stick for scheduled runs?
s

Sushma Adari

06/02/2022, 2:34 PM
The parameter shows under the RUN CONFIG tab. I added it in the code directly under run_config.
k

Kevin Kho

06/02/2022, 2:35 PM
Can I see a screen shot of that just to check it was set right?
s

Sushma Adari

06/02/2022, 2:35 PM
the code or the tab or both?
k

Kevin Kho

06/02/2022, 2:36 PM
Either
s

Sushma Adari

06/02/2022, 2:36 PM
run_config=DockerRun(
              image=IMAGE,
              labels=LABEL,
               env={"PREFECT__CLOUD__HEARTBEAT_MODE": "thread"}
          ),
 spirited-lorikeet OVERVIEWPARAMETERSRUN CONFIG
{
  "env": {
    "PREFECT__CLOUD__HEARTBEAT_MODE": "thread"
  },
  "type": "DockerRun",
  "image": "spiderrockadvisors/py38-base:2.0.1",
  "ports": null,
  "labels": [
    "sradock01"
  ],
  "__version__": "1.2.0",
  "host_config": null
}
k

Kevin Kho

06/02/2022, 2:46 PM
If this still fails, what we normally see is that the container is really dying/crashing. A good test is making this step into it’s own flow and turning off heartbeats and seeing if it runs to completion
s

Sushma Adari

06/02/2022, 2:48 PM
Gotcha, will try that out. Is there a follow-up fix for if it fails within its own Flow?
k

Kevin Kho

06/02/2022, 2:49 PM
Bump up memory or something maybe for the container it runs in
Most often it’s memory related
✅ 1
s

Sushma Adari

06/02/2022, 2:50 PM
Thank you, I will look into it
@Kevin Kho So I have run multiple tests for this issue. The memory is not being overloaded max usage tops out at 180 mb, the timing is not running for very long and the Flow Run is only failing on scheduled runs still. Not sure what the next testing steps are. Still the same issue of 'No heartbeat detected from the remote task; marking the run as failed.'
k

Kevin Kho

06/07/2022, 2:53 PM
Do you know how to reliably replicate? If you do we can definitely explore it more
s

Sushma Adari

06/07/2022, 2:54 PM
Can you explain what that is?
k

Kevin Kho

06/07/2022, 2:55 PM
Like do you have a code snippet we could look at and run to encounter the same issue?
Another thing you can do though is make that database querying step a subflow and turning off heartbeats for the subflow and then calling it with 
create_flow_run
s

Sushma Adari

06/07/2022, 2:56 PM
Hmm, not sure that is viable given the data being used is sensitive information
k

Kevin Kho

06/07/2022, 2:58 PM
Then I think you can either turn off heartbeats or make it a subflow without heartbeats
✅ 1
s

Sushma Adari

06/07/2022, 2:59 PM
I will give it a go, thanks
5 Views
#prefect-communityJoin Slack
Powered by