So i was able to narrow it down to a such case: im...
# prefect-communitya
Alexander
11/20/2020, 12:01 PMSo i was able to narrow it down to a such case: imagine you have a flow with two branches, one branch is a long-running and other is short lived and may fail.
So if short branch fails, its dependencies are triggerfailed. If i re-run failed task in a still running flow, its state is set to pending and nothing happens. Flow finishes when long task fininished and this task is still in pending state.
But, if before re-running failed task i set downstream triggerfailed task to pending state, re-running failed task executes as expected.
Is this a bug?
n
nicholas
11/20/2020, 3:07 PM
Hi @Alexander - can you give us some idea of your setup? For example it'd be helpful to know where you're running your flows/which agent you're using/if you have results configured on your upstream tasks etc.
a
Alexander
11/20/2020, 3:11 PMAh yeah sure, if that helps. I am running prefect server with docker agent and localdaskexecutor.
This is a sample flow i managed to write that seems like reproduces the issue. How to use: touch /tmp/fail and run flow. problem task fails. Then rm /tmp/fail and re-run flow run from failed problem task while long running task is still in progress
Alexander
11/20/2020, 3:12 PMIf you set downstream task from triggerfailed to pending before re-running it will work properly
n
nicholas
11/20/2020, 3:13 PM
Ah ok, and this is on Prefect Server, correct?
nicholas
11/20/2020, 3:14 PM
(that you're re-running the flow)
a
Alexander
11/20/2020, 3:14 PMYes
n
nicholas
11/20/2020, 3:14 PM
Which version of Prefect Core did you spin up Server from?
a
Alexander
11/20/2020, 3:15 PM Copy code
PREFECT_SERVER_TAG=core-0.13.14
PREFECT_APOLLO_TAG=core-0.13.14
PREFECT_UI_TAG=core-0.13.14
PREFECT_AGENT_TAG=0.13.14-python3.7n
nicholas
11/20/2020, 3:16 PM
Great, let me investigate this quickly, thanks @Alexander
❤️ 1
nicholas
11/20/2020, 3:19 PM
Alright! Would you mind upgrading to the latest version of Prefect Core and then restarting your Server?
nicholas
11/20/2020, 3:19 PM
I believe the bug you're experiencing is documented here: https://github.com/PrefectHQ/ui/issues/369 and was resolved in 0.13.15
a
Alexander
11/20/2020, 4:05 PM@nicholas unfortunately issue still there. Using 0.13.16 everywhere.
I started flow, the problem task failed, its downstream triggerfailed, i restarted problem task and it stuck at pending state
Alexander
11/20/2020, 4:07 PMAnd the logs but nothing helpful there i suppose
Alexander
11/20/2020, 4:13 PMBut IF i restart the whole flow run while long task is still running it is successfully clears downstream and problem task states and runs them
n
nicholas
11/20/2020, 4:14 PM
Got it - I'm testing with your flow in just a minute here but how are you restarting the task? I think your issue might be resolved if you use the
Restarta
Alexander
11/20/2020, 4:14 PMBut in this case it marks whole flow run failed even if my failed tasks restarted.
Alexander
11/20/2020, 4:14 PMI use restart button on task run page
n
nicholas
11/20/2020, 4:15 PM
Ok great, then I think you've done what you need to, let me see if there's more we can look at otherwise I'll open a bug ticket
a
Alexander
11/20/2020, 4:16 PMYeah appreciated it, i am here to provide any additional context if needed
🙂 1
n
nicholas
11/20/2020, 5:31 PM
Following up on this, I believe I've identified the problem, which was a UI bug where the task run dialog was failing to correctly identify downstream task runs (and thereby never set their states correctly). Fix here: https://github.com/PrefectHQ/ui/pull/452
🎉 1
4 Views