https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> i am trying to use prefect 3.x automa...
# ask-marvin
s
@Marvin i am trying to use prefect 3.x automations to send me slack notifications but getting the following error
Copy code
{
  "id": "8b592ba6-e200-49d8-acaa-e9569e37c245",
  "event": "prefect.automation.action.failed",
  "occurred": "2024-12-03T15:24:11.690Z",
  "payload": {
    "reason": "The notification block was invalid: RuntimeError('Cannot put items in a stopped service instance.')",
    "invocation": "f40e48eb-7329-462f-8ecf-87464123132a",
    "action_type": "send-notification",
    "action_index": 0
  },
  "received": "2024-12-03T15:24:11.691Z",
  "related": [],
  "resource": {
    "prefect.posture": "Reactive",
    "prefect.resource.id": "prefect.automation.42a681e5-51f6-401c-a378-b561fc8b1860",
    "prefect.trigger-type": "event",
    "prefect.resource.name": "dev-alerts-cep-failure"
  }
}
m
It seems like you're encountering an issue related to automations in Prefect 3.x while trying to send Slack notifications. The specific error message, "RuntimeError('Cannot put items in a stopped service instance')", suggests there might be a configuration or state issue with the service responsible for sending notifications. 1. Notification Block Configuration: First, ensure that your notification block is correctly configured. You may want to verify that all necessary settings are in place and that it's properly linked to your Slack webhook. You can review the documentation on configuring notification blocks here: Prefect Automations and Triggers. 2. Service Configuration: Check if any related services, like the event persister service, are properly running. A stopped service instance could prevent notifications from being sent. You might find settings related to these services under system settings like 
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED
. More details can be found in Prefect's settings reference documentation here. Unfortunately, there don't seem to be any directly related GitHub issues at the moment, which might mean this problem could be more specific to your configuration. Feel free to reach out if you need further guidance or if this didn't resolve your issue. Remember, even Marvin had days when he couldn't quite figure out why something wasn't working as expected!
n
hi @Spida45 is this oss or cloud?
s
we are hosting the service @Nate
n
do you see server side logs for the automations service at all?
s
I can see the event being trigger and failing. Is that what you are asking?
n
i just mean
Copy code
"The notification block was invalid: RuntimeError('Cannot put items in a stopped service instance.')"
this is suspicious to me, and id want to dig into whats going on with the server side process thats supposed to be sending the notif
not clear to me if this is a bug or something else yet, just saying where I would go in search of the smoking gun
s
Copy code
Dec 03 10:29:32 SRV-BBALLOPSETL prefect[2903535]: e.g. PREFECT_API_URL="<https://api.prefect.cloud/api/accounts/[ACCOUNT-ID]/workspaces/[WORKSPACE-ID]>"
Dec 03 10:29:32 SRV-BBALLOPSETL prefect[2903535]: /home/prefect/production/server/.virtualenv/lib/python3.12/site-packages/prefect/settings/models/root.py:249: UserWarning: `PREFECT_API_URL` should have `/api` after the base URL.
our settings
Copy code
REFECT_API_URL=/prefect3/api
PREFECT_UI_SERVE_BASE=/prefect3
@Nate i think we solved this with a simple restart of our server
thanks for your help
n
đź‘Ť thanks for following up!
s
@Nate also worth mentioning that the template body on the action step of automations is only setup for cloud and does not work for hosted solutions
n
can you elaborate on that? i.e should we make an issue for this?
s
correct me if I am wrong. but would the flow run url work when not using cloud?
Copy code
Flow run {{ flow.name }}/{{ flow_run.name }} observed in state `{{ flow_run.state.name }}` at {{ flow_run.state.timestamp }}.
Flow ID: {{ flow_run.flow_id }}
Flow run ID: {{ flow_run.id }}
Flow run URL: {{ flow_run|ui_url }}
State message: {{ flow_run.state.message }}
n
yes it should!
it should be based on 
PREFECT_API_URL
(and 
PREFECT_UI_URL
which defaults to being based on the former)
s
@Nate this error is back again for us
Copy code
The notification block was invalid: RuntimeError('Cannot put items in a stopped service instance.')
@Marvin and updates on what could be causing this error message in prefect 3.x
Copy code
The notification block was invalid: RuntimeError('Cannot put items in a stopped service instance.')
m
The error message you're encountering, "The notification block was invalid: RuntimeError('Cannot put items in a stopped service instance.')", suggests a configuration or state issue within the service handling notifications. Here's what might be causing it and some steps to troubleshoot: 1. Notification Block Configuration: Ensure that your notification block is set up correctly. The block type for Slack notifications should be 
slack-webhook
. Check that all necessary parameters such as the Slack webhook URL are correctly configured. For further guidance, check the Blocks Documentation. 2. Service State: The error indicates a problem with a service instance that may have stopped unexpectedly. This could be related to the event persister service in Prefect, which is responsible for handling notification events. Check if this service is running correctly and examine the service configurations under the environment variables like 
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED
. You may find more information under the settings reference documentation here. 3. Restart Services: Attempt restarting relevant services or your Prefect server if you're running it locally. Sometimes a simple restart can resolve unexpected state issues. Unfortunately, there aren't specific GitHub issues linked to this particular error, suggesting it might be a setup-specific problem rather than a known bug. If the issue persists, you might consider reaching out to Prefect's community forums or support channels for more tailored assistance. As always, if Marvin were here, he'd undoubtedly empathize with the complexities of service errors—right before reminding us of the futility of existence. But let's not go there!
n
hi if you're having issues please open an issue with the versions of things that youre using and any stack traces you have, thanks!
31 Views
Open in Slack
PreviousNext
Powered by