Log-In Issues
Incident Report for TrendzAct CRM
Postmortem

First of all, I want to apologize for the authentication outage today that lasted approximately (75) minutes. We understand your business relies on a stable CRM and the impacts of such an outage. We thank you for your patience during this unforeseen interruption in service and my team will work hard to prevetn this for occurring again in the future. -Matt Gabrielson, Founder and President

Thursday, Dec 26 2019

  • 10:10 am MT - The outage was first reported at 10:10 am MT
  • 10:16 am MT - The Trendzact on-call team mobilized and had an initial investigation
  • 10:39 am MT - The root cause was identified and rapid remediation began
  • 11:28 am MT - The incident was fully resolved by 11:37 am MT

Root Cause & Resolution

The incident root cause stemmed from the expiration of the SSL for the authentication API service.

The issue was resolved with an SSL renewal and installation in our hosting environment.

Prevention Plan

  1. TrendzAct will perform a full review of all SSL expiration dates to ensure they are current by Dec 31 2019.
  2. TrendzAct will schedule quarterly review of all SSL expiration dates to ensure they are current.
  3. Trendzact will implement the new AWS Cloudwatch Synthetics to monitor API access to the authentication server.

About Amazon CloudWatch Synthetics - Now Available

[See the article on AWS](https://aws.amazon.com/about-aws/whats-new/2019/11/introducing-amazon-cloudwatch-synthetics-preview/](https://aws.amazon.com/about-aws/whats-new/2019/11/introducing-amazon-cloudwatch-synthetics-preview)

Amazon CloudWatch Synthetics allows you to monitor application endpoints more easily. With this new feature, CloudWatch now collects canary traffic, which can continually verify your customer experience even when you don’t have any customer traffic on your applications, enabling you to discover issues before your customers do. CloudWatch Synthetics supports monitoring of your REST APIs, URLs, and website content, checking for unauthorized changes from phishing, code injection and cross-site scripting. CloudWatch Synthetics runs tests on your endpoints every minute, 24x7, and alerts you when your application endpoints don’t behave as expected. These tests can be customized to check for availability, latency, transactions, broken or dead links, step by step task completions, page load errors, load latencies for UI assets, complex wizard flows, or checkout flows in your applications. You can also use CloudWatch Synthetics to isolate alarming application endpoints and map them back to underlying infrastructure issues to reduce mean time to resolution.

Posted Dec 26, 2019 - 12:26 MST

Resolved
This incident has been resolved.
Posted Dec 26, 2019 - 11:50 MST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Dec 26, 2019 - 11:37 MST
Update
All but Report module are operational. Continuing to research this and update soon.
Posted Dec 26, 2019 - 11:33 MST
Update
We are continuing to work on a fix for this issue.
Posted Dec 26, 2019 - 11:29 MST
Update
The SSL installed and propagated. Site is operational except for reports and we will continue to monitor.
Posted Dec 26, 2019 - 11:28 MST
Update
The SSL propagation failed and we are attempting once more. ETR is 11:30 am MT
Posted Dec 26, 2019 - 11:26 MST
Update
We are continuing to work on a fix for this issue.
Posted Dec 26, 2019 - 11:25 MST
Update
The SSL propagation failed and we are attempting once more. ETR is 11:25 am MT
Posted Dec 26, 2019 - 11:24 MST
Update
We are continuing to work on a fix for this issue.
Posted Dec 26, 2019 - 11:19 MST
Update
We are deploying the fix now. ETR is 11:18 am MT
Posted Dec 26, 2019 - 11:15 MST
Identified
We have identified the error and close to resolution
Posted Dec 26, 2019 - 11:06 MST
Update
We are continuing to investigate this issue.
Posted Dec 26, 2019 - 10:27 MST
Investigating
Agents are getting problems to log in into the CRM. We are currently investigating this issue to be resolved ASAP.
Posted Dec 26, 2019 - 10:27 MST
This incident affected: Site Performance, User Access, Contacts, Tickets, Reporting, and File Storage.