https://prefect.io logo
r

Romain

02/18/2022, 7:21 AM
Hello folks, Is it possible to have a run with some tasks with status TriggerFailed, while there is no upstream Failed tasks? See screenshot below:
n

Noah Holm

02/18/2022, 8:46 AM
Do you mean as a hypothetical question, or is your screenshot real? If the screenshot is real then I guess the answer is yes?
😄 1
r

Romain

02/18/2022, 8:55 AM
It s real 🙂 . I saw this thread https://prefect-community.slack.com/archives/CL09KU1K7/p1642622235321000 So it seems I m not the only one to see such a behavior. But I m not really sure I understand the solution they provided there
n

Noah Holm

02/18/2022, 9:06 AM
I suppose you’re seeing the same error message? Never heard of this before myself but it seems like they solved it by moving this part
Copy code
with Flow("Using Collections") as flow:
    a = a_number()
    b = a_number()
    s = get_sum([a, b])
to this
Copy code
with Flow("Using Collections") as flow:
    a = a_number()
    b = a_number()
    a_list = merge_into_list(a, b)
    s = get_sum(a_list)
could you be doing something similar to the top code with your merge tasks for example?
upvote 1
r

Romain

02/18/2022, 9:45 AM
I saw the suggested solution, but I don't really get why this should help in my case. I don't think I have any non task input for this TriggerFailed task. What I can't really get is if the error is : "Trigger was "all_successful" but some of the upstream tasks failed." I should then being able to find in the prefect server which upstream task failed right? But when I do so , I can see none.
n

Noah Holm

02/18/2022, 10:18 AM
Yeah regardless if you’re doing the same thing with e.g., collection args for this TriggerFailed task it seems like the error message is a bit misleading. If you’ve verified that you’re not doing the same as above I suppose there’s another issue here - does this happen every time? Could there be something else with result serialization maybe?
a

Anna Geller

02/18/2022, 10:27 AM
@Romain this page explains the issue and shows how you can fix it. LMK if this solution doesn't work for you
r

Romain

02/18/2022, 10:46 AM
@Anna Geller I saw this example in the thread mentioned above. But here is the main difference with what I observe: in your example we can see task_fail_a and task_fail_b having status
Failed
. Right?
a

Anna Geller

02/18/2022, 10:53 AM
Yes. Can you share your flow?
r

Romain

02/18/2022, 10:55 AM
Sadly I can't (because of IP) .
But if we analyze the flow given in this thread, All the upstream tasks are also successful right?
a

Anna Geller

02/18/2022, 11:01 AM
Can you build a small reproducible example? This would be even better. I won't be able to help unless I can see the flow, you can also DM it to me
r

Romain

02/18/2022, 11:12 AM
Yes I understand that. I m trying to find a way to reproduce it, but not sure how. It's not something I've seen often. One thing that I wonder whether it could play a role: In our system (prefect-server deployed on k8s), we are seeing quite often some issue with Apollo (see screenshot). I have no idea why but i suspect our Hasura pod is the problem here (I can see it getting restarted very often by k8s). So here is my question: could this prefect server "infra" error have any consequences on the flow run itself?
a

Anna Geller

02/18/2022, 11:19 AM
No. The trigger error you see would be the same even if you would execute it with Prefect Cloud. If you are looking for a quick-fix to run this task, you can assign a trigger
all_finished
- this way, this task will run even if some of those upstream tasks fails.
6 Views