Tomás Emilio Silva Ebensperger

    Tomás Emilio Silva Ebensperger

    1 year ago
    Hi everyone. I am having trouble with timing out flows. Some of them reach the set timeout but they ignore them. Now i have even set up an automation to cancel the flow on the UI, and it does cancel it, but it is stuck at "canceling". I have to manually set the state to canceled to free the slot. any clues?
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Tomás Emilio Silva Ebensperger, are you using Dask as the executor here? Is the flow hanging due to a lack of resources?
    Tomás Emilio Silva Ebensperger

    Tomás Emilio Silva Ebensperger

    1 year ago
    LocalAgent
    Kevin Kho

    Kevin Kho

    1 year ago
    Are you doing some kind of API call? Or is everything processed by the agent?
    Tomás Emilio Silva Ebensperger

    Tomás Emilio Silva Ebensperger

    1 year ago
    Api call, (salesforce) it takes a while to return but when run on my machine for example, it returns... i set the timeout at 15 minutes (even a timeout on the requests session), but it doesn't work
    Kevin Kho

    Kevin Kho

    1 year ago
    Ah can you show me a small example? I think there might be something off with the setup
    Tomás Emilio Silva Ebensperger

    Tomás Emilio Silva Ebensperger

    1 year ago
    of course thank you!
    handler = slack_notifier(only_states=[Failed], ignore_states=[TriggerFailed])
    task_time_out = 600
    
    @task(log_stdout=True, state_handlers=[handler], timeout=task_time_out)
    def extract(client_instance, limit=None):
    	client_instance.extract_all(limit=limit)
    I use composition there.
    i treated for this the task as a wrapper of the main class, that connects to Salesforce and the correspoinding DB
    from prefect.agent.local import LocalAgent
    
    
    if __name__ == '__main__':
    	agent = LocalAgent()
    	agent.start()
    and runs on a virtual machine and it is has been successful so far
    Kevin Kho

    Kevin Kho

    1 year ago
    Does the Salesforce client have a timeout built-in also? You’re saying this setup worked on the small example? But not with the full code?
    Tomás Emilio Silva Ebensperger

    Tomás Emilio Silva Ebensperger

    1 year ago
    it is a requests call. i set the timeout there and also on the prefect side as you can see. then also the automation on the UI. when i run the flow on the machine manually the timeouts seem to work fine... even they run fine with the cloud agent whren the timeouts are not too long the issue seems to happen when the timeouts are longer than 10 minutes for example
    the UI logs says the tasks keeps running
    Kevin Kho

    Kevin Kho

    1 year ago
    When you set the timeout in the request call, are you doing it inside a task? I think the issue is that the request library is synchronous so it waits for a response and the timeout is unable to stop it (same thing with cancellation being blocked). If you set the requests timeout outside of a task, it is applied during registration but not during runtime so that would be the thing to check.
    Tomás Emilio Silva Ebensperger

    Tomás Emilio Silva Ebensperger

    1 year ago
    i am going to try that, thank you
    but it is weird, would this affect the timeout set on the prefect task? the weird thing is that it fails when the timeout is set for too long, like it gets dizzy
    Kevin Kho

    Kevin Kho

    1 year ago
    No but the
    timeout
    on Prefect is not guaranteed, it’s really a best effort because it’s very hard to stop an ongoing Python process gracefully. It’s more reliable when the compute is all on the same machine than when you start a process in Kubernetes or Dask. In this case though, I actually have seen timeouts worked on API calls (boto3), but it was coupled with a timeout on the boto3 client. I am not super sure about the Prefect timeout stopping API calls so my hope is the client stops it and then Prefect can exit the task
    Tomás Emilio Silva Ebensperger

    Tomás Emilio Silva Ebensperger

    1 year ago
    thank you kevin i appreciate the help