No news is bad news
Today I encountered an issue with AWS Chatbot. Chatbot is the AWS service I use to send triggered CloudWatch alarms into Slack. Unlike email, people might actually read them there.
Issues with Chatbot aren’t unusual. I’ve been using it since it was in beta*, and have encountered some frankly weird quirks. However, today was different. A quick Google of the error message returned the result that makes every DevOps Engineers heart sink.
*AWS Chatbot is now GA and much more stable.
How did I get here?
Like so many people, I got furloughed at the start of April due to the Covid-19 pandemic.
During furlough there were 0 alerts in Slack for outages. If you don’t test, the numbers look great, right…
On my first day back, I headed into CloudWatch to look for some VPC flow logs, only to see an angry looking red ALARM staring at me from CloudWatch.
“Hmm” I thought, “I didn’t get any alerts, wtf.”
So off to Chatbot I go. The first thing I do is try to edit the Chatbot workspace, and was promptly told to go piss up a rope as soon as I clicked it.
AWS Chatbot can’t retrieve the list of public channels right now. To continue configuring this channel, enter the channel ID or the channel URL.
After digging through CloudWatch, SNS, Slack docs, AWS docs and poking a doll of Werner Vogels with cocktail sticks, I was no closer to resolving this. I also still hadn’t got my VPC flow logs.
Then it hit me. My Slack account had been used to authorise AWS Chatbot into Slack. The same account that had been disabled during furlough. Even though it was reenabled, this wasn’t enough to get the alerts flowing again.
You’ve talked enough shit, tell me how to fix it.
You need to go into Chatbot and look for “Configure new client.”
Then choose Slack and hit “Configure”.
