RevRouter-Bandwidth call routing issue
Incident Report for TrendzAct CRM
Postmortem

Scott & Connect Your Leads, I an truly sorry for the Bandwidth-RevRouter API service interruption that caused the call routing outage between 11:45 am MT (Dec 26) and 01:55 am MT (Dec 27). We know you rely on RevRouter to answer, screen and transfer you inbound calls and we failed to so during the outage and for that we sincerly apologize. -Matt Gabrielson, Founder and President

Timeline

Thursday, Dec 26 2019

  • 06:52 pm MT - The outage was first reported
  • 06:59 pm MT - The Trendzact on-call team mobilized and had an initial investigation
  • 07:29 pm MT - Investigation with Bandwidth support indicated the received a 200 OK
  • 10:49 pm MT - The TrendzAct dev team accepted the ticket and began remediation

Friday, Dec 27 2019

  • 01:55 am MT - The root cause was identified and resolution was complete
  • 05:32 am MT - The incident was fully monitored and resolved

Root Cause & Resolution

The incident root cause was the AWS server for Bandwidth API had its IIS service in an stopped state.

The issue was resolved by restarting the IIS service.

Prevention Plan

  1. Trendzact will implement the new AWS Cloudwatch Synthetics to monitor several API commands to simulate access to the Bandwidht API server.

About Amazon CloudWatch Synthetics - Now Available

[See the article on AWS]([https://aws.amazon.com/about-aws/whats-new/2019/11/introducing-amazon-cloudwatch-synthetics-preview/](https://aws.amazon.com/about-aws/whats-new/2019/11/introducing-amazon-cloudwatch-synthetics-preview)](https://aws.amazon.com/about-aws/whats-new/2019/11/introducing-amazon-cloudwatch-synthetics-preview)

Posted Dec 27, 2019 - 08:13 MST

Resolved
This incident has been resolved.
Posted Dec 27, 2019 - 04:13 MST
Monitoring
We have monitored the results. RevRouter-Bandwidth calls work.
Posted Dec 27, 2019 - 04:10 MST
Identified
The root cause was identified and fixed.
Posted Dec 27, 2019 - 04:04 MST
Update
OnCall team has been notified and activated. Confirmed Bandwidth getting 200 Answer and reviewing RTR engine.
Posted Dec 26, 2019 - 22:50 MST
Investigating
We are currently investigating this issue.
Posted Dec 26, 2019 - 22:47 MST
This incident affected: API and External Access (RevRouter-Bandwidth).