https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I have set an up an automation using ...
# ask-marvin
g
@Marvin I have set an up an automation using the prefect web UI to notify me via slack when a state enters an error state. If I constrain it to a resource (eg a work queue), there are never any events that are emitted, even though I can see them in the Events page. Is this intentional or a bug?
m
✅ thought for 33.6 seconds
This behavior where events aren't being emitted when constrained to a resource like a work queue does sound unexpected. While the documentation doesn't explicitly address this scenario, it should theoretically work since automations are designed to respond to events across different scopes of resources. I'd recommend creating a GitHub issue to report this behavior, as it appears to be either a bug or at least a case that needs clarification in the documentation. In the meantime, here are a few suggestions: 1. As a workaround, you could try creating the automation without the resource constraint and instead add a condition in the trigger that checks for the work queue or resource ID in the event data. 2. Double-check that the work queue you're monitoring is actively receiving and processing work - though if you can see the events in the Events page, this is likely not the issue. 3. Verify that the automation's trigger configuration exactly matches the event type you're seeing in the Events page. Would you mind sharing: 1. The specific trigger configuration you're using 2. An example of the events you're seeing in the Events page This would help in creating a more detailed bug report and potentially finding a more specific workaround for your use case.
g
@Marvin The trigger is:
Copy code
{
  "type": "event",
  "match": {
    "prefect.resource.id": [
      "prefect.work-queue.9376b608-5918-4d90-a4c8-64ebfb913312"
    ]
  },
  "match_related": {
    "prefect.resource.role": "deployment"
  },
  "after": [],
  "expect": [
    "prefect.flow-run.TimedOut",
    "prefect.flow-run.Cancelled",
    "prefect.flow-run.Failed",
    "prefect.flow-run.Cancelling",
    "prefect.flow-run.Crashed"
  ],
  "for_each": [
    "prefect.resource.id"
  ],
  "posture": "Reactive",
  "threshold": 1,
  "within": 0
}
An example event that I expected to be matched by this trigger, but was not, is:
Copy code
{
  "id": "0197ed7b-c021-79ae-9916-986a70f93c57",
  "account": "9b6bc821-21ac-425c-9a22-ddb1e1df5d59",
  "event": "prefect.task-run.Failed",
  "occurred": "2025-07-09T04:39:59.777Z",
  "payload": {
    "intended": {
      "from": "RUNNING",
      "to": "FAILED"
    },
    "initial_state": {
      "type": "RUNNING",
      "name": "Running",
      "message": "",
      "state_details": {}
    },
    "validated_state": {
      "type": "FAILED",
      "name": "Failed",
      "message": "Task run encountered an exception PolygonOutOfBoundsError",
      "state_details": {
        "retriable": false
      },
      "data": null
    },
    "task_run": {
      "name": "check_polygons_within_boundary-af7",
      "task_key": "check_polygons_within_boundary-539bf9ba",
      "dynamic_key": "af70ba9d-46a9-468b-ba55-d39d31ae1312",
      "empirical_policy": {
        "max_retries": 0,
        "retry_delay_seconds": 0,
        "retries": 0,
        "retry_delay": 0
      },
      "tags": [],
      "labels": {},
      "task_inputs": {
        "polygons": [
          {
            "input_type": "task_run",
            "id": "0197ed7b-be43-7b6f-95c6-e234f75950c3"
          },
          {
            "input_type": "task_run",
            "id": "0197ed7b-bf5c-7973-8ade-ae7c1a813776"
          }
        ],
        "project_boundary": [
          {
            "input_type": "task_run",
            "id": "0197ed7b-bf5c-7973-8ade-ae7c1a813776"
          }
        ]
      },
      "run_count": 1,
      "flow_run_run_count": 1,
      "expected_start_time": "2025-07-09T04:39:59.737920Z",
      "start_time": "2025-07-09T04:39:59.761357Z",
      "end_time": "2025-07-09T04:39:59.777696Z",
      "total_run_time": 0.016339
    }
  },
  "received": "2025-07-09T04:39:59.882Z",
  "related": [
    {
      "prefect.resource.id": "prefect.flow-run.0686df29-d9b2-746c-8000-713f0222ae4c",
      "prefect.resource.role": "flow-run",
      "prefect.resource.name": "ccu_v2_flow-CCP397/ccu_evidence_flow"
    },
    {
      "prefect.resource.id": "prefect.flow.87b4ea72-7c1a-4ec6-9059-ca20df1f6872",
      "prefect.resource.role": "flow",
      "prefect.resource.name": "ccu_evidence_flow"
    }
  ],
  "resource": {
    "prefect.resource.id": "prefect.task-run.0197ed7b-bff9-7c7f-99e0-62468bd572ca",
    "prefect.resource.name": "check_polygons_within_boundary-af7",
    "prefect.state-message": "Task run encountered an exception PolygonOutOfBoundsError",
    "prefect.state-name": "Failed",
    "prefect.state-timestamp": "2025-07-09T04:39:59.777696+00:00",
    "prefect.state-type": "FAILED",
    "prefect.orchestration": "client"
  },
  "workspace": "28c0ee61-44f8-4ac0-802a-d15b833d1c28"
}
m
✅ thought for 21.1 seconds
Based on the event data and research, here's what's happening: 1. The task-run failure event doesn't contain the work queue ID in either its 
resource
or 
related
fields. This is expected behavior - work queue information is only available at the flow-run level. 2. To properly catch both task and flow failures for a specific work queue, you'll need to: a. Either modify your trigger to watch for both task and flow events b. Or rely on flow-run failure events only (since a flow run will typically fail when its tasks fail) I recommend option (b) as it's cleaner and will avoid duplicate notifications. Your current trigger configuration is correct for this approach - you just need to ensure that: 1. The work queue ID is correctly associated with the deployment 2. The flow is configured to fail when tasks fail (this is the default behavior) If you want to verify the work queue association, you can check the deployment details in the UI or use the CLI command:
Copy code
bash
prefect deployment inspect <deployment-name>
This will show you which work queue the deployment is using and confirm whether it matches the ID in your trigger configuration.
g
@Marvin Yep, you're right. The event is a task failure rather than a flow failure. Here is a flow failure that I would expect to show up:
Copy code
{
  "id": "0686df2a-0511-7f44-8000-89a2ce4d2680",
  "account": "9b6bc821-21ac-425c-9a22-ddb1e1df5d59",
  "event": "prefect.flow-run.Failed",
  "occurred": "2025-07-09T04:40:00.316Z",
  "payload": {
    "intended": {
      "from": "RUNNING",
      "to": "FAILED"
    },
    "initial_state": {
      "type": "RUNNING",
      "name": "Running"
    },
    "validated_state": {
      "type": "FAILED",
      "name": "Failed",
      "message": "Flow run encountered an exception: PolygonOutOfBoundsError"
    }
  },
  "received": "2025-07-09T04:40:00.446Z",
  "related": [
    {
      "prefect.resource.id": "prefect.flow.4b57357e-7e47-40d2-be82-4ca942170e94",
      "prefect.resource.role": "flow",
      "prefect.resource.name": "ccu_v2_flow"
    },
    {
      "prefect.resource.id": "prefect.deployment.3e1ed183-9f82-4976-9f21-49bdb7bd028a",
      "prefect.resource.role": "deployment",
      "prefect.resource.name": "prod",
      "prefect.deployment.version-id": "0685b40d-837e-7a64-8000-4b12162e6529",
      "prefect.deployment.version-type": "vcs:bitbucket",
      "prefect.deployment.version": "e41de1ad"
    },
    {
      "prefect.resource.id": "prefect.work-queue.9376b608-5918-4d90-a4c8-64ebfb913312",
      "prefect.resource.role": "work-queue",
      "prefect.resource.name": "prod"
    },
    {
      "prefect.resource.id": "prefect.work-pool.3c008ba7-129a-4ae3-813a-284d524d2fd5",
      "prefect.resource.role": "work-pool",
      "prefect.resource.name": "default-worker-pool-prefect-v3",
      "prefect.work-pool.type": "ecs"
    },
    {
      "prefect.resource.id": "prefect.tag.CCP397",
      "prefect.resource.role": "tag"
    },
    {
      "prefect.resource.id": "prefect.tag.ccu_v2_flow",
      "prefect.resource.role": "tag"
    },
    {
      "prefect.resource.id": "prefect.tag.prod",
      "prefect.resource.role": "tag"
    },
    {
      "prefect.resource.id": "prefect-cloud.user.029e9a92-833a-44e7-b3c0-f8b16abf3c40",
      "prefect.resource.role": "creator",
      "prefect.resource.name": "ml-teamcarboncropnz"
    }
  ],
  "resource": {
    "prefect.resource.id": "prefect.flow-run.0686df11-21f0-7a7e-8000-ea8a126b7c53",
    "prefect.resource.name": "ccu_v2_flow-CCP397",
    "prefect.state-message": "Flow run encountered an exception: PolygonOutOfBoundsError",
    "prefect.state-name": "Failed",
    "prefect.state-timestamp": "2025-07-09T04:40:00.316910+00:00",
    "prefect.state-type": "FAILED"
  },
  "workspace": "28c0ee61-44f8-4ac0-802a-d15b833d1c28"
}
m
✅ thought for 52.3 seconds
Based on the event data and research, I can see a potential issue with your trigger configuration. While the work queue ID is present in the 
related
array, your trigger might not be matching it correctly. Let's try modifying your trigger configuration slightly:
Copy code
json
{
  "type": "event",
  "match": {
    "prefect.resource.id": [
      "prefect.work-queue.9376b608-5918-4d90-a4c8-64ebfb913312"
    ]
  },
  "match_related": {
    "prefect.resource.role": "work-queue"  // Changed from "deployment" to "work-queue"
  },
  "expect": [
    "prefect.flow-run.TimedOut",
    "prefect.flow-run.Cancelled",
    "prefect.flow-run.Failed",
    "prefect.flow-run.Cancelling",
    "prefect.flow-run.Crashed"
  ],
  "for_each": [
    "prefect.resource.id"
  ],
  "posture": "Reactive",
  "threshold": 1,
  "within": 0
}
The key changes: 1. Changed 
match_related.prefect.resource.role
to 
"work-queue"
since we're trying to match based on the work queue 2. Kept the work queue ID in the 
match
section If this still doesn't work, there's one more variation we could try:
Copy code
json
{
  "type": "event",
  "match": {},
  "match_related": {
    "prefect.resource.id": "prefect.work-queue.9376b608-5918-4d90-a4c8-64ebfb913312",
    "prefect.resource.role": "work-queue"
  },
  "expect": [
    "prefect.flow-run.TimedOut",
    "prefect.flow-run.Cancelled",
    "prefect.flow-run.Failed",
    "prefect.flow-run.Cancelling",
    "prefect.flow-run.Crashed"
  ],
  "for_each": [
    "prefect.resource.id"
  ],
  "posture": "Reactive",
  "threshold": 1,
  "within": 0
}
This moves the work queue matching entirely into the 
match_related
section, which might be more appropriate since the work queue appears in the 
related
array of the event. Could you try one of these modified configurations and let me know if it resolves the issue? Also, it might be worth creating a GitHub issue about this behavior, as it seems like the documentation could be clearer about how 
match
and 
match_related
interact with resources in the event's 
related
array.
g
@Marvin Is there anywhere I can see what events are emitted by which resource?
m
✅ thought for 60.4 seconds
Based on the research, there are a few ways to understand what events are emitted by which resources: 1. Events Page in the UI: The most practical way is to use the Events page in the Prefect UI. You can filter by resource type and observe what events are emitted. This gives you real-world examples of events in your environment. 2. Documentation Reference: The official documentation at https://docs-3.prefect.io/v3/concepts/events lists the common event types, though it may not be exhaustive. 3. Programmatic Exploration: You can use the Prefect API to query events. Here's a Python snippet that would help:
Copy code
python
from prefect.client import get_client

async with get_client() as client:
    # Get events for the last hour
    events = await client.read_events(
        limit=100,
        event_filter={"occurred": {"after": "1h"}}
    )
    
    # Group events by resource type
    event_types_by_resource = {}
    for event in events:
        resource_type = event.resource.type
        event_type = event.event
        if resource_type not in event_types_by_resource:
            event_types_by_resource[resource_type] = set()
        event_types_by_resource[resource_type].add(event_type)
Common event types include: 1. Flow Run Events: - 
prefect.flow-run.Completed
- 
prefect.flow-run.Failed
- 
prefect.flow-run.Running
- 
prefect.flow-run.Scheduled
- 
prefect.flow-run.Pending
- 
prefect.flow-run.Crashed
- 
prefect.flow-run.Cancelling
- 
prefect.flow-run.Cancelled
2. Task Run Events: - 
prefect.task-run.Completed
- 
prefect.task-run.Failed
- 
prefect.task-run.Running
3. Deployment Events: - 
prefect.deployment.Created
- 
prefect.deployment.Updated
4. Work Queue Events: - 
prefect.work-queue.Paused
- 
prefect.work-queue.Resumed
5. Work Pool Events: - 
prefect.work-pool.Created
- 
prefect.work-pool.Updated
- 
prefect.work-pool.Deleted
The best practice for creating automations is to: 1. First observe the specific events you want to react to in the UI's Events page 2. Note the exact event names and resource structures 3. Create your automation trigger based on these observed events This ensures you're working with actual events that exist in your system rather than theoretical ones that might not be emitted. Would you like me to provide more specific details about any particular resource type's events?
g
@Marvin I have changed my trigger to:
Copy code
{
  "type": "event",
  "match": {
    "prefect.resource.id": [
      "prefect.work-queue.9376b608-5918-4d90-a4c8-64ebfb913312"
    ]
  },
  "match_related": {
    "prefect.resource.role": "work-queue"
  },
  "after": [],
  "expect": [
    "prefect.flow-run.Cancelled",
    "prefect.flow-run.Crashed",
    "prefect.flow-run.TimedOut",
    "prefect.flow-run.Failed",
    "prefect.flow-run.Cancelling"
  ],
  "for_each": [
    "prefect.resource.id"
  ],
  "posture": "Reactive",
  "threshold": 1,
  "within": 0
}
But it is still not seeing any events
m
✅ thought for 20.1 seconds
Looking at your event data and the trigger configuration, I think I see the issue. The problem is in how we're trying to match the work queue. Let's analyze the flow run failure event you shared earlier: 1. The work queue appears in the 
related
array as:
Copy code
json
{
  "prefect.resource.id": "prefect.work-queue.9376b608-5918-4d90-a4c8-64ebfb913312",
  "prefect.resource.role": "work-queue",
  "prefect.resource.name": "prod"
}
2. The flow run itself is the primary resource in the event, not the work queue. This means we need to modify our approach. Try this modified trigger configuration:
Copy code
json
{
  "type": "event",
  "match": {},
  "match_related": {
    "prefect.resource.id": [
      "prefect.work-queue.9376b608-5918-4d90-a4c8-64ebfb913312"
    ]
  },
  "expect": [
    "prefect.flow-run.TimedOut",
    "prefect.flow-run.Cancelled",
    "prefect.flow-run.Failed",
    "prefect.flow-run.Cancelling",
    "prefect.flow-run.Crashed"
  ],
  "for_each": [
    "prefect.resource.id"
  ],
  "posture": "Reactive",
  "threshold": 1,
  "within": 0
}
Key changes: 1. Moved the work queue ID matching entirely to 
match_related
since it's in the 
related
array 2. Removed the 
prefect.resource.role
constraint as it might be redundant (the ID already specifies it's a work queue) 3. Empty 
match
object since we're not matching against the primary resource If this still doesn't work, we could try one more variation that matches against the flow run ID pattern:
Copy code
json
{
  "type": "event",
  "match": {
    "prefect.resource.id": "prefect.flow-run.*"
  },
  "match_related": {
    "prefect.resource.id": [
      "prefect.work-queue.9376b608-5918-4d90-a4c8-64ebfb913312"
    ]
  },
  "expect": [
    "prefect.flow-run.TimedOut",
    "prefect.flow-run.Cancelled",
    "prefect.flow-run.Failed",
    "prefect.flow-run.Cancelling",
    "prefect.flow-run.Crashed"
  ],
  "for_each": [
    "prefect.resource.id"
  ],
  "posture": "Reactive",
  "threshold": 1,
  "within": 0
}
This explicitly states we're looking for flow run events (
prefect.flow-run.*
) that are related to our specific work queue. Could you try these configurations and let me know if either works? Also, it might be worth checking the automation's history in the UI to see if it's receiving the events but not triggering for some other reason.
Open in Slack
PreviousNext
Powered by