No news is bad news
Today I encountered an issue with AWS Chatbot. Chatbot is the AWS service I use to send triggered CloudWatch alarms into Slack. Unlike email, people might actually read them there.
Issues with Chatbot aren’t unusual. I’ve been using it since it was in beta*, and have encountered some frankly weird quirks. However, today was different. A quick Google of the error message returned the result that makes every DevOps Engineers heart sink.
*AWS Chatbot is now GA and much more stable.
How did I get here?
Like so many people, I got furloughed at the start of April due to the Covid-19 pandemic.
During furlough there were 0 alerts in Slack for outages. If you don’t test, the numbers look great, right…
On my first day back, I headed into CloudWatch to look for some VPC flow logs, only to see an angry looking red ALARM staring at me from CloudWatch.
“Hmm” I thought, “I didn’t get any alerts, wtf.”
So off to Chatbot I go. The first thing I do is try to edit the Chatbot workspace, and was promptly told to go piss up a rope as soon as I clicked it.
AWS Chatbot can’t retrieve the list of public channels right now. To continue configuring this channel, enter the channel ID or the channel URL.
After digging through CloudWatch, SNS, Slack docs, AWS docs and poking a doll of Werner Vogels with cocktail sticks, I was no closer to resolving this. I also still hadn’t got my VPC flow logs.
Then it hit me. My Slack account had been used to authorise AWS Chatbot into Slack. The same account that had been disabled during furlough. Even though it was reenabled, this wasn’t enough to get the alerts flowing again.
You’ve talked enough shit, tell me how to fix it.
You need to go into Chatbot and look for “Configure new client.”
Then choose Slack and hit “Configure”.
Next, you will see AWS Chatbot asking for permission to access your Slack workspace. Allow it.
After this, you will not need to do any further configuration. You should start getting your alerts back into Slack again. You can test this by editing one of your alarms to some ridiculously low value which will trigger if you even look at it.
How do I prevent this so I never need to read this drivel again?
This occurred because I used my own account to link Slack and AWS. As a meatbag, I’m fragile and prone to failure, and so is my Slack account. The way to prevent this is to create an account which you only use for configuring Slack alerts in Chatbot. This account, since it never belongs to a person, won’t suddenly quit to go live in the woods and break all the alerts.
Once you do that, you will need to repeat this process when you’re logged in with your Slack service account.
Reauthenticate Slack and ChatBot to fix this. In future, create a service account and don’t use an account for meatbags. I never got those flow logs.