Based on the research and available information, here's what we know and what you should check:
1. Check the Event Feed First
- In Prefect Cloud, go to the Event Feed and look for events with the type 'prefect-cloud.webhook.failed'
- This will show if the webhooks are being triggered but failing to deliver
2. Verify Webhook Configuration
- Check if the webhook is properly configured in Prefect Cloud
- Verify that the service account assigned to the webhook (if any) has the correct permissions
3. Check Receiving System
- The 15-minute delay suggests there might be an issue with the system receiving the webhooks
- Check if the receiving system has any rate limiting, queuing, or processing delays
- Verify that the receiving system's logs show when it actually receives the webhook
4. Network and Infrastructure
- Check for any network latency issues between Prefect Cloud and your receiving system
- Verify there are no proxy or firewall rules causing delays
To debug this effectively, I recommend:
1. First, confirm in the Prefect Cloud Event Feed that the webhook events are being triggered immediately when the flow run status changes
2. Check the webhook configuration in Prefect Cloud for any error events
3. Add logging on your receiving system to timestamp exactly when the webhook is received
4. Compare the timestamps between:
- When the flow run status changes
- When Prefect Cloud sends the webhook (from Event Feed)
- When your system receives the webhook
Could you check the Event Feed and let me know if you see any webhook failure events or if the timestamps there match what you're expecting? This will help narrow down where in the chain the delay is occurring.