Platform temporarily unavailable
Incident Report for Reply.ai
Postmortem

Reply performs maintenance operations on its infrastructure regularly. These interventions are required to provide new features, fix issues and improve the overall system performance. Some of these operations require changes to the database, which can result in a few seconds of downtime to guarantee data integrity.

From time to time, changes to the database require significantly more time to be applied. In order to maintain application availability, these operations are prepared by the team to be run concurrently —ie. Without affecting the normal behavior of the application.

Reply v1.5.8, to be released on Tuesday, November 13th 2018, modified the way bot messages are stored in the database. Given the high volume of messages in Reply, this is a long-running operation. However, we failed to flag this intervention and did not take necessary measures to ensure availability. When the release was published and the operation started, the database prevented any data access until all modifications were complete.

All releases are closely monitored by the team. When an unusual running time was detected, the team announced an outage and started investigating. After the problem was identified, the team started verifying possible solutions that would guarantee no data loss. When a solution was finally applied, the release was rolled back and the system availability verified.

Posted 5 months ago. Nov 14, 2018 - 04:52 EST

Resolved
We have identified and resolved the connection issue affecting the service.
Posted 5 months ago. Nov 13, 2018 - 09:50 EST
Update
We are currently investigating connection problems to our database.
Working to restore the service as soon as possible.
Posted 5 months ago. Nov 13, 2018 - 09:39 EST
Identified
We have identified a problem that affects platform availability. We are currently working on restoring the service. Sorry for the inconvenience.
Posted 5 months ago. Nov 13, 2018 - 08:35 EST
This incident affected: US region (reply.ai) (API, Web App, Message delivery, Analytics).