https://prefect.io logo
Title
f

Florian Guily

06/28/2022, 10:10 AM
Hi everyone, prefect 1 question here. I deployed an ETL flow and i need to keep track of what happened during each flow run such as "how many record have been ingested, how many failed etc". Ideally this would be done at the end of the flow execution so i looked about flow state handlers in the doc. But i couldn't find if i could use "task" data in those state handlers. Do you have any idea ?
e

emre

06/28/2022, 1:17 PM
Things that come to my mind: 1- you can create an artifact at the end of each flow run. This may work if you only want to inspect manually and one by one. 2- flow constructor apparently has a
terminal_state_handler
. It receives the flow state, but also the terminal task states which should contain task results. 3- I think
flow.state
object contains every single task state in a dict as well.
👍 1
f

Florian Guily

06/28/2022, 1:21 PM
Thanks for your reply, i saw the existence of
terminal_state_handler
but couldn't find any doc about it. Seems to correspond to what i need based on what you said. Btw, how do Prefect determine the terminal task if there are multiple tasks with no children ?
e

emre

06/28/2022, 1:24 PM
by default, every leaf task (task with no children) are reference tasks, and are passed to
terminal_state_handler
. You can explicitly set terminal (reference) tasks via
flow.set_reference_tasks(task_list)
🙌 1
At least I believe so, I never used
terminal_state_handler
😅
f

Florian Guily

06/28/2022, 1:28 PM
ok i see thanks !