https://prefect.io
Join Slack
I have what may be a dumb question. I've set my tasks to have multiple retries. I've noticed that if...
# ask-community
j
I have what may be a dumb question. I've set my tasks to have multiple retries. I've noticed that if the first try fails, the others will always fail as well, no matter how long the delay. Example - I lose connection to our database during an extract task. 2 minutes pass before a retry, and it loses the connection again. A day later (next run), things go fine. So, are the retries actually doing anything?
k
They most certainly do, but maybe not if it was an upstream task that returned a corrupted connection right? The upstream task won’t retry. Do you have this kind of setup
j
No, the connection is fine, its more of erroring out while running the extract.
Runs fine sometimes, doesn't run at others. But not matter what it won't run successfully if it fails once
k
Could you show me the task definition?
j
Copy code
@task(name = "Extract", log_stdout=True, checkpoint=False, max_retries=3, retry_delay=timedelta(seconds=10))
def extract(connection: any, start_date: str) -> any:
    df=pd.read_sql_query(f"""
    select bla bla 
    """, con=connection)
    return df
k
Are you using a class or the decorator?
The only thing I can think of here is that the 
connection
is somehow not working to begin with. If you’d like to test if retries do anything, you can try adding logging to a broken piece of code.
j
Is there a way to re-establish the connection then? So if the extract fails, loop back and re-try the connection. My guess is right now it loses the connection and then just retries the extract with the (now lost) connection, because the connection was technically already successful.
k
Unfortunately not unless you couple them in the same task so the retry hits both.
Orion will let you treat this as a functional unit and retry all
j
How far off is orion haha
so just combine tasks, see how that does? Makes sense honestly
k
End of feb is when it will be off technical preview
j
Great, thanks for the idea. I'm going to try that out.
👍 1
6 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by