https://prefect.io logo
Channels
Powered by
t

Thomas Opsomer

03/04/2022, 4:22 PM
Hello community 🙂 We experienced a strange behaviour of the "restart" button on prefect cloud: When going to a task run and restarting the task, it correctly reschedules the task, but it doesn't reschedule the downstream tasks (in the UI they stay green instead of light blue)
k

Kevin Kho

03/04/2022, 4:25 PM
Only Failed tasks are re-scheduled with the restart button. We know there are cases where you want to re-run upstream/downstream so for Orion (Prefect 2.0) this will be supported first-class with subflows being logical units that you can restart together
t

Thomas Opsomer

03/04/2022, 4:34 PM
Ah ok. So the fact that it used to work was just luck ? 🙂 Then maybe you should update the message in the UI when clicking the restart button to specify that only the failed/not_successful downstream dependents will be re-run
k

Kevin Kho

03/04/2022, 4:45 PM
Ah ok we can add a ticket in the UI repo for that
Do you know what current one was?
t

Thomas Opsomer

03/04/2022, 4:47 PM
yep:
Click on confirm to restart the flow run _flow_run_name_ from this task run. This will restart your flow run and re-run task _task_name_ and its downstream dependents.
k

Kevin Kho

03/04/2022, 4:47 PM
Thank you!
@Marvin open “Edit Message of Restart Button for More Clarity” in UI
m

Marvin

03/04/2022, 4:48 PM
t

Thomas Opsomer

03/07/2022, 8:35 AM
Hello @Kevin Kho, to follow on this topic: Given what you told me, I manually put to "failed" the downstream tasks that I want to re-run, however prefect only restart the first task, see the screenshot.
k

Kevin Kho

03/07/2022, 2:00 PM
How did you manually mark them as failed? Through the UI?
I can try this in a bit and if it fails, we might need to use caching to get the behavior right.
I tested this and it worked for me. Here is my Flow:
Copy code
from prefect import task, Flow
import prefect
import datetime

@task
def task_one():
    <http://prefect.context.logger.info|prefect.context.logger.info>(datetime.datetime.now())
    return 1

@task
def task_two(x):
    <http://prefect.context.logger.info|prefect.context.logger.info>(datetime.datetime.now())
    return x+1

@task
def task_three(x):
    <http://prefect.context.logger.info|prefect.context.logger.info>(datetime.datetime.now())
    return x+1

@task
def task_four(x):
    <http://prefect.context.logger.info|prefect.context.logger.info>(datetime.datetime.now())
    return x+1

with Flow("fail_propagate") as flow:
    task_four(task_three(task_two(task_one())))
I ran this. It was all successful and then i marked tasks 2, 3 and 4 as failed. I then marked the flow run as failed and restarted. Let me see what happens if I don’t mark the flow run as failed
It behaves the same whether or not I mark the Flow run as Failed. Looking more
My current thinking right now is that even if I can replicate the behavior you see, a fix won’t be so straightforward. I would suggest you try caching maybe and then you can start a new flow run without repeating the work. Have you seen caching ? You just need to invalidate the cache when you want to re-run
t

Thomas Opsomer

03/07/2022, 5:51 PM
• Yes I marked them as Failed using the UI • I haven't looked at caching yet, will do thanks 🙂 • Something else that I noticed, after marking the tasks as failed: ◦ if a restart the flow from the task (so using the restart button on the task-run page) then it doesn't work ◦ if a restart the flow from the flow-run page and pressing restart, it works correctly
k

Kevin Kho

03/07/2022, 5:53 PM
Ah yeah because I think restarting from the task run page is just for the task, and from the flow page is for the Flow
You should be good then right?
t

Thomas Opsomer

03/07/2022, 6:25 PM
yep, but if this behaviour is expected, the message in the UI is again misleading 🙂
k

Kevin Kho

03/07/2022, 7:18 PM
yeah the issue for that has been opened
Powered by