hey all. Question about the `INTERNAL_SERVER_ERROR` and prefect's secret client throwing the `KeyErr...

Andrew Pruchinski

09/30/2022, 4:46 PM

hey all. Question about the

INTERNAL_SERVER_ERROR

and prefect's secret client throwing the

KeyError: 'SLACK_WEBHOOK_URL'

. I know there is a current intermittent issue in regards to retrieving secrets. We pass our secrets to the dask worker via the docker container so we aren't using the prefect secrets stored in the UI. However we are still getting this error and it has brought down multiple flows this morning. The error, other information and questions will be in the threads. Thank you in advance!

1️⃣ 1

✅ 1

Andrew Pruchinski

09/30/2022, 4:47 PM

The stack trace

Andrew Pruchinski

09/30/2022, 4:49 PM

We do pass the

slack_notifier

as a decorator to our tasks to notify failures in our slack channel. Does this use the prefect secret in the UI behind the scenes and could be the reason we are seeing this error?

Andrew Pruchinski

09/30/2022, 4:50 PM

We set up some automations when a flow enters in a success or failed state, it uses the

SLACK_WEBHOOK_URL

in the Prefect Secrets to send this message to our slack channel. Would this bring down the entire flow though?

Jeff Hale

09/30/2022, 8:35 PM

Hi Andrew, we just deployed a fix that should alleviate the secret issue. So hopefully that is not a problem anymore, please let us know if it is. To answer your question, yes, I think a failure in the flow from the secret could have been the reason for the error.

Andrew Pruchinski

09/30/2022, 9:15 PM

thank you @Jeff Hale for the response! I will let you know if we see this error again. have a great weekend

🙏 1

Jeff Hale

09/30/2022, 9:17 PM

You too!

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.