https://prefect.io
Join Slack
So i was able to narrow it down to a such case: imagine you have a flow with two branches, one branc...
# ask-community
a
So i was able to narrow it down to a such case: imagine you have a flow with two branches, one branch is a long-running and other is short lived and may fail. So if short branch fails, its dependencies are triggerfailed. If i re-run failed task in a still running flow, its state is set to pending and nothing happens. Flow finishes when long task fininished and this task is still in pending state. But, if before re-running failed task i set downstream triggerfailed task to pending state, re-running failed task executes as expected. Is this a bug?
n
Hi @Alexander - can you give us some idea of your setup? For example it'd be helpful to know where you're running your flows/which agent you're using/if you have results configured on your upstream tasks etc.
a
Ah yeah sure, if that helps. I am running prefect server with docker agent and localdaskexecutor. This is a sample flow i managed to write that seems like reproduces the issue. How to use: touch /tmp/fail and run flow. problem task fails. Then rm /tmp/fail and re-run flow run from failed problem task while long running task is still in progress
debug.py
If you set downstream task from triggerfailed to pending before re-running it will work properly
n
Ah ok, and this is on Prefect Server, correct?
(that you're re-running the flow)
a
Yes
n
Which version of Prefect Core did you spin up Server from?
a
Copy code
PREFECT_SERVER_TAG=core-0.13.14
PREFECT_APOLLO_TAG=core-0.13.14
PREFECT_UI_TAG=core-0.13.14
PREFECT_AGENT_TAG=0.13.14-python3.7
n
Great, let me investigate this quickly, thanks @Alexander
❤️ 1
Alright! Would you mind upgrading to the latest version of Prefect Core and then restarting your Server?
I believe the bug you're experiencing is documented here: https://github.com/PrefectHQ/ui/issues/369 and was resolved in 0.13.15
a
@nicholas unfortunately issue still there. Using 0.13.16 everywhere. I started flow, the problem task failed, its downstream triggerfailed, i restarted problem task and it stuck at pending state
And the logs but nothing helpful there i suppose
logs (1).txt
But IF i restart the whole flow run while long task is still running it is successfully clears downstream and problem task states and runs them
n
Got it - I'm testing with your flow in just a minute here but how are you restarting the task? I think your issue might be resolved if you use the 
Restart
button on the task run page itself, rather than manually setting the state
a
But in this case it marks whole flow run failed even if my failed tasks restarted.
I use restart button on task run page
n
Ok great, then I think you've done what you need to, let me see if there's more we can look at otherwise I'll open a bug ticket
a
Yeah appreciated it, i am here to provide any additional context if needed
🙂 1
n
Following up on this, I believe I've identified the problem, which was a UI bug where the task run dialog was failing to correctly identify downstream task runs (and thereby never set their states correctly). Fix here: https://github.com/PrefectHQ/ui/pull/452
🎉 1
8 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by