Hi, after upgrading from 3.1.5 to 3.1.15 my Deplo...
# ask-community
t
Hi, after upgrading from 3.1.5 to 3.1.15 my Deployments that are triggered by multiple events (compound require all triggers) do no longer trigger. When the automation should kick in instead the following message is logged by the prefect-server:
Message failed after 4 retries and will be moved to the dead letter queue
I'm using server and worker helm chart found here: https://github.com/PrefectHQ/prefect-helm This happens for all my compound triggers.
Where can I find the dead letter queue? Looked in the PostgreSQL Prefect DB but found nothing.
n
hi @Thomas Eckestad - hmm this sounds like it could be a bug. can you share roughly your setup so I can try to reproduce?
t
Hi, I've set together an example that you can run locally. Follow the instructions in the
local/README.md
and you will have the example up and running within a few minutes.
n
hi @Thomas Eckestad for security reasons i'm not going to unzip that on my machine. feel free to put the detail on public github if you can or simply into an issue
t
I see, no problem. I have put the code here: https://github.com/thoec/prefect-hello-world
🙏 1
Any update on this @Nate? Would you like me to create an issue instead?
n
hi thomas! sorry for the delay - I have reproduced this using your repo
Copy code
09:43:36.279 | DEBUG   | prefect.server.utilities.messaging.memory - Retried message on topic='actions' retry_count=1 queue_size=0 retry_queue_size=1
09:43:36.279 | DEBUG   | prefect.server.utilities.messaging.memory - Retried message on topic='actions' retry_count=2 queue_size=0 retry_queue_size=1
09:43:36.280 | DEBUG   | prefect.server.utilities.messaging.memory - Retried message on topic='actions' retry_count=3 queue_size=0 retry_queue_size=1
09:43:36.280 | WARNING | prefect.server.utilities.messaging.memory - Message failed after 4 retries and will be moved to the dead letter queue
though its not totally clear to me why this is happening, I suspect it might have to do with some bug in the actions service related to compound triggers, so if you could open an issue that would be helpful!
cc @Chris Guidry (just in case you see something funny here)
c
Sorry I don't see anything right off the bat, but I'm surprised no other error is being logged there if the
actions
service is kicking it back
hard same 1
t
I looked deeper into this since we are heavily relying on event-driven workflows. I created the following PR with a suggested solution to the problem: https://github.com/PrefectHQ/prefect/pull/17356. Wdyt?
👍 1