https://prefect.io logo
n

Newskooler

10/21/2020, 3:13 PM
Hi 👋 🙂 Is it possible to set a task which already ran successfully as RETRY from a job down the stream? My situation is this: I have data which i want to write to a DB. To do this I have 3 tasks for the DB: •
open_db_connection
(run only once during a flow) •
save_to_db
run multiple times during one flow •
close_db_connection
- run once at the end of the flow Sometimes when I
save_to_db
the connection has dropped out (for various reasons). Would a clean solution be to set the current
save_to_db
task as
RETRY
and then set one of the tasks it depends on (the
open_db_connection
task) as RETRY too. This way it will (or so I expect) to open a db connection first and then retry the failed task again. 🤔 is my thinking correct and is that possible or it’s a messy solution? Thanks : )
s

Spencer

10/21/2020, 3:25 PM
I'd put it all in the
save_to_db
task. It'll make a connection, save the data, then close the connection. I think it can be a bit dicey to try to manage shared resources outside the flow tasks and share them across tasks rather than having each task isolated.
n

Newskooler

10/21/2020, 3:27 PM
Okay, thanks. So I was only worried if there are multiple workers, whether this approach will fail or not 🤔
j

Jeremiah

10/21/2020, 3:39 PM
Indeed, passing DB connections around will only work if you are guaranteed that all your tasks exist in a shared memory space (with a compatible executor) - if you have multiple workers, you won’t be able to pass the connection between them. I agree with @Spencer that combining them will be more effective.
👍 2
n

Newskooler

10/21/2020, 3:40 PM
Thanks both, I will try that then:)