https://prefect.io logo
Join Slack
Powered by
Hi everyone. I am having trouble with timing out f...
# prefect-server
t
Hi everyone. I am having trouble with timing out flows. Some of them reach the set timeout but they ignore them. Now i have even set up an automation to cancel the flow on the UI, and it does cancel it, but it is stuck at "canceling". I have to manually set the state to canceled to free the slot. any clues?
k
Hey @Tomás Emilio Silva Ebensperger, are you using Dask as the executor here? Is the flow hanging due to a lack of resources?
t
LocalAgent
k
Are you doing some kind of API call? Or is everything processed by the agent?
t
Api call, (salesforce) it takes a while to return but when run on my machine for example, it returns... i set the timeout at 15 minutes (even a timeout on the requests session), but it doesn't work
k
Ah can you show me a small example? I think there might be something off with the setup
t
of course thank you!
Copy code
handler = slack_notifier(only_states=[Failed], ignore_states=[TriggerFailed])
task_time_out = 600

@task(log_stdout=True, state_handlers=[handler], timeout=task_time_out)
def extract(client_instance, limit=None):
	client_instance.extract_all(limit=limit)
I use composition there.
i treated for this the task as a wrapper of the main class, that connects to Salesforce and the correspoinding DB
Copy code
from prefect.agent.local import LocalAgent


if __name__ == '__main__':
	agent = LocalAgent()
	agent.start()
and runs on a virtual machine and it is has been successful so far
k
Does the Salesforce client have a timeout built-in also? You’re saying this setup worked on the small example? But not with the full code?
t
it is a requests call. i set the timeout there and also on the prefect side as you can see. then also the automation on the UI. when i run the flow on the machine manually the timeouts seem to work fine... even they run fine with the cloud agent whren the timeouts are not too long the issue seems to happen when the timeouts are longer than 10 minutes for example
the UI logs says the tasks keeps running
k
When you set the timeout in the request call, are you doing it inside a task? I think the issue is that the request library is synchronous so it waits for a response and the timeout is unable to stop it (same thing with cancellation being blocked). If you set the requests timeout outside of a task, it is applied during registration but not during runtime so that would be the thing to check.
t
i am going to try that, thank you
but it is weird, would this affect the timeout set on the prefect task? the weird thing is that it fails when the timeout is set for too long, like it gets dizzy
k
No but the 
timeout
on Prefect is not guaranteed, it’s really a best effort because it’s very hard to stop an ongoing Python process gracefully. It’s more reliable when the compute is all on the same machine than when you start a process in Kubernetes or Dask. In this case though, I actually have seen timeouts worked on API calls (boto3), but it was coupled with a timeout on the boto3 client. I am not super sure about the Prefect timeout stopping API calls so my hope is the client stops it and then Prefect can exit the task
t
thank you kevin i appreciate the help
👍 1
3 Views
Open in Slack
PreviousNext
Powered by