https://prefect.io logo
Channels
Powered by
a

Alexander

11/20/2020, 12:01 PM
So i was able to narrow it down to a such case: imagine you have a flow with two branches, one branch is a long-running and other is short lived and may fail. So if short branch fails, its dependencies are triggerfailed. If i re-run failed task in a still running flow, its state is set to pending and nothing happens. Flow finishes when long task fininished and this task is still in pending state. But, if before re-running failed task i set downstream triggerfailed task to pending state, re-running failed task executes as expected. Is this a bug?
n

nicholas

11/20/2020, 3:07 PM
Hi @Alexander - can you give us some idea of your setup? For example it'd be helpful to know where you're running your flows/which agent you're using/if you have results configured on your upstream tasks etc.
a

Alexander

11/20/2020, 3:11 PM
Ah yeah sure, if that helps. I am running prefect server with docker agent and localdaskexecutor. This is a sample flow i managed to write that seems like reproduces the issue. How to use: touch /tmp/fail and run flow. problem task fails. Then rm /tmp/fail and re-run flow run from failed problem task while long running task is still in progress
debug.py
If you set downstream task from triggerfailed to pending before re-running it will work properly
n

nicholas

11/20/2020, 3:13 PM
Ah ok, and this is on Prefect Server, correct?
(that you're re-running the flow)
a

Alexander

11/20/2020, 3:14 PM
Yes
n

nicholas

11/20/2020, 3:14 PM
Which version of Prefect Core did you spin up Server from?
a

Alexander

11/20/2020, 3:15 PM
Copy code
PREFECT_SERVER_TAG=core-0.13.14
PREFECT_APOLLO_TAG=core-0.13.14
PREFECT_UI_TAG=core-0.13.14
PREFECT_AGENT_TAG=0.13.14-python3.7
n

nicholas

11/20/2020, 3:16 PM
Great, let me investigate this quickly, thanks @Alexander
❤️ 1
Alright! Would you mind upgrading to the latest version of Prefect Core and then restarting your Server?
I believe the bug you're experiencing is documented here: https://github.com/PrefectHQ/ui/issues/369 and was resolved in 0.13.15
a

Alexander

11/20/2020, 4:05 PM
@nicholas unfortunately issue still there. Using 0.13.16 everywhere. I started flow, the problem task failed, its downstream triggerfailed, i restarted problem task and it stuck at pending state
And the logs but nothing helpful there i suppose
logs (1).txt
But IF i restart the whole flow run while long task is still running it is successfully clears downstream and problem task states and runs them
n

nicholas

11/20/2020, 4:14 PM
Got it - I'm testing with your flow in just a minute here but how are you restarting the task? I think your issue might be resolved if you use the 
Restart
button on the task run page itself, rather than manually setting the state
a

Alexander

11/20/2020, 4:14 PM
But in this case it marks whole flow run failed even if my failed tasks restarted.
I use restart button on task run page
n

nicholas

11/20/2020, 4:15 PM
Ok great, then I think you've done what you need to, let me see if there's more we can look at otherwise I'll open a bug ticket
a

Alexander

11/20/2020, 4:16 PM
Yeah appreciated it, i am here to provide any additional context if needed
🙂 1
n

nicholas

11/20/2020, 5:31 PM
Following up on this, I believe I've identified the problem, which was a UI bug where the task run dialog was failing to correctly identify downstream task runs (and thereby never set their states correctly). Fix here: https://github.com/PrefectHQ/ui/pull/452
🎉 1
3 Views
Powered by