https://prefect.io logo
Powered by
Title
v

Vaibhav Ariyur

11/22/2021, 3:00 PM
shown left: I run a flow, flow enters subflow, completes successfully, and enters the "clean" task successfully when there is a constantly running "keep-alive" task running in parallel. The keep-alive task just logs "keeping alive" every 10 seconds for an indefinite period of time. shown right: I run a flow, flow enters subflow, subflow completes successfully but the main flow does not understand that (hence the grey status for the subflow), main flow restarts instead of entering the clean task (hence the greyed out box in one of the prior tasks --> where I cancelled this flow). Anybody understand why this happens, and a non-hackish fix for this?
k

Kevin Kho

11/22/2021, 3:03 PM
Hey @Vaibhav Ariyur, how did you call with subflow? 
StartFlowRun
or 
create_flow_run
?
v

Vaibhav Ariyur

11/22/2021, 3:10 PM
subflow= StartFlowRun(flow_name="subflow run", project_name="Tests", wait=True)

with Flow("flow run", storage=Azure(container="prefect-flows", connection_string=AZURE_CONNECT_STRING)) as flow:
    Param=Parameter("Param")
    task1 = task_1(Param)
    task2=task_2()
    task3=task_3(task1,task2,Param)
    task4=subflow(upstream_tasks=[task3], parameters={"Subflow param 1":"some_param", "Subflow param 2": "some_param_2"})
	task4_5=keep_alive(task3)
    task5=clean(task4)
I used StartFlowRun
k

Kevin Kho

11/22/2021, 3:13 PM
Just to make sure I understand, the parent flow triggered a subflow, but the subflow was cancelled, and the parent doesn’t know the subflow was cancelled. And then restarting the parent flow enters that subflow (but nothing happens) and doesn’t proceed to the 
clean
task right?
v

Vaibhav Ariyur

11/22/2021, 3:18 PM
The parent flow triggered a subflow --> yes subflow was cancelled --> No, subflow completed successfully, but the parent did not acknowledge that the subflow completed succesfully. Instead the parent flow just restarts when it should be acknowledging the success and going to the next task And then the restarting parent flow enters that subflow... --> No, the restarting parent flow is on task1 and I manually cancel it.
think of it this way: pre-subflow tasks --> subflow task --> pre-subflow tasks (without keep-alive) pre-subflow tasks --> subflow task --> clean (with keep-alive)
k

Kevin Kho

11/22/2021, 4:58 PM
could you try the 
wait_for_flow_run
with 
raise_final_state=True
? This is available in Prefect 0.15.9 so you may need to upgrade.
:upvote: 1
z

Zanie

11/22/2021, 5:27 PM
Are you getting any particular message when your main flow restarts?
v

Vaibhav Ariyur

11/23/2021, 8:53 PM
Sorry for the late reply, was taking a bit of a while to upgrade to 0.5.19. After I figured it out, I ran with 
wait_for_flow_run
 and 
raise_final_state=True
still getting the same issue
and @Zanie I do not get any particular message
z

Zanie

11/23/2021, 9:03 PM
What do you mean by “restarts” then? Are there some state changes and logs?
v

Vaibhav Ariyur

11/23/2021, 9:22 PM
^^Already completed steps are rerunning
any ideas on this?
once again, this is on prefect k8s server and not on prefect cloud, and PREFECT___CLOUD___HEARTBEAT_MODE="thread" is not solving this
a

Anna Geller

11/24/2021, 8:58 PM
@Vaibhav Ariyur I think we would have to try reproduce this. Can you try build a minimal example we can try, and share your “prefect diagnostics” so that we know on which version we should check it? It’s hard to tell what is the issue here. You have a parent flow and from that you trigger some child flows. What got cancelled? Did you cancel some flow run manually? Is any of the flow runs in questions executed on Dask? Sorry for so many questions but would need more info to help here.
2 Views
#prefect-serverJoin Slack
Powered by