Hi, a newbie question… If the trigger for all task...
# prefect-community
j
Hi, a newbie question… If the trigger for all tasks in a flow is “all successful”, and one of the upstream task fails, all of the downstream tasks are “trigger failed” (even if the upstream failed task is restarted and succeeds). This forces me to restart the trigger-failed tasks one by one. Is this the intended behavior? or am I missing something? Should I change the trigger to “all finished” in order to avoid this manual restarting of the downstream tasks when one of the upstream tasks fails at least once?
c
Hi Jasono, the easiest way to achieve this is to use the “Restart” button in the UI on the Flow Run page. If this is not an option for you, then you can set all trigger failed states to
Pending
and then reschedule the flow run to handle all trigger failed tasks simultaneously. I don’t recommend changing the trigger to
all_finished
unless your tasks don’t depend on the upstream succeeding and don’t depend on any data exchange between the tasks
j
I tried the “Restart” button, and the status of the first trigger failed task changed to “Pending”, but then the status of the flow changes back to “Failed” from “Scheduled”
The “restarted” task’s status stays as “Pending”.
c
Correct - I might be misunderstanding your question if that’s not what you were looking for
They will rerun once your agent resubmits the run
j
How do I make the agent resubmit the run? Or does it automatically do it?
Why is resubmission required? I would think clicking the “Restart” button restarts the flow from the failed tasks.
c
It sounds like you might benefit from a tutorial on how prefect works: I recommend https://docs.prefect.io/orchestration/tutorial/configure.html and https://docs.prefect.io/orchestration/agents/overview.html
j
Sorry, I scanned through these docs again but they don’t answer my question.
For instance, is the resubmission of a restarted flow mentioned in these docs?
c
Flows only run when agents submit them for execution; restarting the flow puts the flow back into a state that allows your agent to rerun it
j
so does the agent automatically rerun it?
Please feel free to direct me to the page where it is discussed in the docs.
c
The agent overview discusses what runs agent picks up and what configuration determines whether an agent will run a flow or not. Assuming you were already running an agent for the first run of this flow then yea the agent will automatically pick it back up whenever you hit the “restart” button
j
Okay. I will wait, but it’s been 30 minutes since the task changed to Pending status, and there is no sign of it restarting anytime soon. I checked the Agent, and it seems to be still on.
I’m running on the Prefect Cloud, and that’s the only flow currently active.
c
Is the run in a scheduled state? It should be picked up immediately. You should check the labels as well (the UI should alert you if there are no agents available with the appropriate labels)
j
No, Failed state.
c
Did you hit the “restart” button?
It should place the run back into a scheduled state, which is the state it needs to be in for the agent to “see” it. You can also manually put it into a scheduled state
j
yes, about 3 times at an interval.
When I hit the restart button, the state changes to Schedule, but then changes again to Failed.
c
Check your logs. It should tell you more about what’s going on - it sounds like your agent is failing the run
j
c
Sorry this will be too hard to debug in a slack thread - would you mind opening a GitHub issue describing your flow and what you tried here? It does seem like something fishy might be going on but hard to know for sure without more info
j
Understood. I may have found a clue.
In the dashboard, it indicates an earlier run as being in progress.
I cancelled (by hitting the cancel button on UI) it hours ago, but the dashboard still indicates it’s running.
I think this might be why Prefect cloud may not restart the other flow.
Is there a way this cancelling task can be killed?
c
Interesting, yea you can use the “set state” Button on any run page (task or flow) and put the run into some finished state
j
It’s working now. Thank you!!
c
Ah excellent! Glad you figured it out 😁
j
I’m coming from Luigi. There is a lot more to learn here 🙂