https://prefect.io logo
Join Slack
Powered by
Hi I have a flow represented by the DAG below. I r...
# ask-community
a
Hi I have a flow represented by the DAG below. I ran it locally with 
flow_result = flow.run()
and visualized it with state. The log says 
Succeed: all reference tasks succeeded
but I can see a failure right there in the stateful DAG. Also it doesn't show the full DAG with the downstream tasks!!! I have also attached the output of 
flow_result.result
and it doesn't show the 
get_duplicated_pairs_info
task. Any tips on what went wrong and how I can debug from here? (Full log in comment)
Copy code
[2021-09-10 16:50:58-0400] INFO - prefect.FlowRunner | Beginning Flow run for 'OVP_step2_dev'
[2021-09-10 16:50:58-0400] INFO - prefect.TaskRunner | Task 'step1_folder_path': Starting task run...
[2021-09-10 16:50:58-0400] INFO - prefect.TaskRunner | Task 'step1_folder_path': Finished task run for task with final state: 'Success'
[2021-09-10 16:50:58-0400] INFO - prefect.TaskRunner | Task 'create_file_path': Starting task run...
[2021-09-10 16:50:58-0400] INFO - prefect.TaskRunner | Task 'create_file_path': Finished task run for task with final state: 'Success'
[2021-09-10 16:50:58-0400] INFO - prefect.TaskRunner | Task 'load_file': Starting task run...
[2021-09-10 16:50:58-0400] INFO - prefect.TaskRunner | Task 'load_file': Finished task run for task with final state: 'Success'
[2021-09-10 16:50:58-0400] INFO - prefect.TaskRunner | Task 'n_iterations': Starting task run...
[2021-09-10 16:50:58-0400] INFO - prefect.TaskRunner | Task 'n_iterations': Finished task run for task with final state: 'Success'
[2021-09-10 16:50:58-0400] INFO - prefect.TaskRunner | Task 'total_iterations': Starting task run...
[2021-09-10 16:50:58-0400] INFO - prefect.TaskRunner | Task 'total_iterations': Finished task run for task with final state: 'Success'
[2021-09-10 16:50:58-0400] INFO - prefect.TaskRunner | Task 'output_folder_path': Starting task run...
[2021-09-10 16:50:58-0400] INFO - prefect.TaskRunner | Task 'output_folder_path': Finished task run for task with final state: 'Success'
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'create_file_path': Starting task run...
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'create_file_path': Finished task run for task with final state: 'Success'
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'GWAS_id': Starting task run...
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'GWAS_id': Finished task run for task with final state: 'Success'
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'outside_id': Starting task run...
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'outside_id': Finished task run for task with final state: 'Success'
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'create_file_path': Starting task run...
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'create_file_path': Finished task run for task with final state: 'Success'
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'load_file': Starting task run...
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'load_file': Finished task run for task with final state: 'Success'
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'resolve_output_folder_path': Starting task run...
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'resolve_output_folder_path': Finished task run for task with final state: 'Success'
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'load_file': Starting task run...
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'load_file': Finished task run for task with final state: 'Success'
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'get_pairs_for_step2': Starting task run...
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'get_pairs_for_step2': Finished task run for task with final state: 'Success'
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'get_duplicated_pair_info': Starting task run...
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'get_duplicated_pair_info': Finished task run for task with final state: 'Failed'
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'template_summary_df_target': Starting task run...
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'template_summary_df_target': Finished task run for task with final state: 'Mapped'
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'template_summary_df_target[0]': Starting task run...
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'template_summary_df_target[0]': Finished task run for task with final state: 'Success'
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'template_summary_df_target[1]': Starting task run...
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'template_summary_df_target[1]': Finished task run for task with final state: 'Success'
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'template_summary_df_target[2]': Starting task run...
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'template_summary_df_target[2]': Finished task run for task with final state: 'Success'
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'create_file_path': Starting task run...
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'create_file_path': Finished task run for task with final state: 'Mapped'
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'create_file_path[0]': Starting task run...
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'create_file_path[0]': Finished task run for task with final state: 'Success'
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'create_file_path[1]': Starting task run...
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'create_file_path[1]': Finished task run for task with final state: 'Success'
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'create_file_path[2]': Starting task run...
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'create_file_path[2]': Finished task run for task with final state: 'Success'
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'load_file': Starting task run...
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'load_file': Finished task run for task with final state: 'Mapped'
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'load_file[0]': Starting task run...
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'load_file[0]': Finished task run for task with final state: 'Success'
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'load_file[1]': Starting task run...
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'load_file[1]': Finished task run for task with final state: 'Success'
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'load_file[2]': Starting task run...
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'load_file[2]': Finished task run for task with final state: 'Success'
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'get_permute_odds_ratio_object': Starting task run...
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'get_permute_odds_ratio_object': Finished task run for task with final state: 'Mapped'
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'get_permute_odds_ratio_object[0]': Starting task run...
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'get_permute_odds_ratio_object[0]': Finished task run for task with final state: 'Success'
[2021-09-10 16:50:59-0400] INFO - prefect.TaskRunner | Task 'get_permute_odds_ratio_object[1]': Starting task run...
[2021-09-10 16:51:00-0400] INFO - prefect.TaskRunner | Task 'get_permute_odds_ratio_object[1]': Finished task run for task with final state: 'Success'
[2021-09-10 16:51:00-0400] INFO - prefect.TaskRunner | Task 'get_permute_odds_ratio_object[2]': Starting task run...
[2021-09-10 16:51:00-0400] INFO - prefect.TaskRunner | Task 'get_permute_odds_ratio_object[2]': Finished task run for task with final state: 'Success'
[2021-09-10 16:51:00-0400] INFO - prefect.TaskRunner | Task 'perform_permutation': Starting task run...
[2021-09-10 16:51:00-0400] INFO - prefect.TaskRunner | Task 'perform_permutation': Finished task run for task with final state: 'Mapped'
[2021-09-10 16:51:00-0400] INFO - prefect.TaskRunner | Task 'process_report_df': Starting task run...
[2021-09-10 16:51:00-0400] INFO - prefect.TaskRunner | Task 'process_report_df': Finished task run for task with final state: 'Mapped'
[2021-09-10 16:51:00-0400] INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded
h
"reference tasks" are what determine the result state
upvote 1
make sure that the tasks you added as dependent tasks are reference tasks
k
You’re saying there are tasks below 
get_duplicated_pair_info
?
a
@Henry thanks for the reminder. I am aware that you can change what the reference tasks are. In this case, I just used the default and did not specified reference tasks. In this case I think the outermost leaf tasks should be the reference tasks.
@Kevin Kho yes it can be seen in the first image of build-time DAG
k
Ah gotcha. What is your executor here?
a
local executor. running the python script that has the flow definition
Found the bug! (kinda, all tasks succeed but it does not do what I want), but am still very interested in why the visualization + logging were inaccurate
k
I wouldn’t know immediately unfortunately 😅
2 Views
Open in Slack
PreviousNext
Powered by