https://prefect.io logo
Channels
Powered by
j

Jason Motley

01/24/2022, 6:11 PM
I have what may be a dumb question. I've set my tasks to have multiple retries. I've noticed that if the first try fails, the others will always fail as well, no matter how long the delay. Example - I lose connection to our database during an extract task. 2 minutes pass before a retry, and it loses the connection again. A day later (next run), things go fine. So, are the retries actually doing anything?
k

Kevin Kho

01/24/2022, 6:14 PM
They most certainly do, but maybe not if it was an upstream task that returned a corrupted connection right? The upstream task won’t retry. Do you have this kind of setup
j

Jason Motley

01/24/2022, 6:15 PM
No, the connection is fine, its more of erroring out while running the extract.
Runs fine sometimes, doesn't run at others. But not matter what it won't run successfully if it fails once
k

Kevin Kho

01/24/2022, 6:17 PM
Could you show me the task definition?
Are you using a class or the decorator?
j

Jason Motley

01/24/2022, 6:18 PM
Copy code
@task(name = "Extract", log_stdout=True, checkpoint=False, max_retries=3, retry_delay=timedelta(seconds=10))
def extract(connection: any, start_date: str) -> any:
    df=pd.read_sql_query(f"""
    select bla bla 
    """, con=connection)
    return df
k

Kevin Kho

01/24/2022, 6:21 PM
The only thing I can think of here is that the 
connection
is somehow not working to begin with. If you’d like to test if retries do anything, you can try adding logging to a broken piece of code.
j

Jason Motley

01/24/2022, 6:22 PM
Is there a way to re-establish the connection then? So if the extract fails, loop back and re-try the connection. My guess is right now it loses the connection and then just retries the extract with the (now lost) connection, because the connection was technically already successful.
k

Kevin Kho

01/24/2022, 6:23 PM
Unfortunately not unless you couple them in the same task so the retry hits both.
Orion will let you treat this as a functional unit and retry all
j

Jason Motley

01/24/2022, 6:24 PM
How far off is orion haha
so just combine tasks, see how that does? Makes sense honestly
k

Kevin Kho

01/24/2022, 6:25 PM
End of feb is when it will be off technical preview
j

Jason Motley

01/24/2022, 6:27 PM
Great, thanks for the idea. I'm going to try that out.
👍 1
6 Views
Powered by