Postmortem -
Read details
Jul 28, 13:04 MDT
Resolved -
Confirmed all sites have been restored.
Jul 28, 12:33 MDT
Update -
We are continuing to monitor for any further issues.
Jul 28, 12:11 MDT
Monitoring -
AWS
[10:49 AM PDT] We continue to see recovery of EC2 instances that were affected by the loss of power in the US-EAST-2 Region. At this stage, the vast majority of affected EC2 instances and EBS volumes have returned to a healthy state and we continue to work on the remaining EC2 instances and EBS volumes. Elastic Load Balancing has shifted traffic away from the affected Availability Zone. Power has been restored to all affected resources and remains stable. We expect the recovery of EC2 instances and EBS volumes to continue to improve over the next 45 minutes.
Jul 28, 12:06 MDT
Update -
AWS confirms this outage and is resolving it.
Trendzact is continuing business continuity plans in case the AWS fix is not successful.
Jul 28, 11:43 MDT
Identified -
from AWS
[10:25 AM PDT] We can confirm that some instances within a single Availability Zone (USE2-AZ1) in the US-EAST-2 Region have experienced a loss of power. Power has been restored to the affected facility and at this stage the majority of the affected EC2 instances have recovered. We expect to recover the vast majority of EC2 instances within the next hour.
Jul 28, 11:32 MDT
Update -
We are continuing to investigate this issue.
Jul 28, 11:21 MDT
Update -
AWS has identified an issue with the Trendzact's primary Region. Trendzact is beginning the business continuity plan for migtation into another AWS region.
Network Connectivity
[10:11 AM PDT] AWS-We are investigating network connectivity issues for some instances and increased error rates and latencies for the EC2 APIs within the US-EAST-2 Region.
https://status.aws.amazon.com/
Jul 28, 11:20 MDT
Investigating -
We are currently investigating this issue.
Jul 28, 11:14 MDT