On Dec 21, 2022 the Rabbet platform experienced a major outage for a total time of 2h34m. During this time no data was lost, nor was any security breached, nor private information compromised. The Rabbet team sincerely apologizes for this disruption of service.

The root cause was an internal networking issue that our cloud hosting provider experienced, which ultimately caused some internal DNS to fail to resolve internal IP addresses between our main API server and one of its data stores.

As the problem began, any of our already-running instances of our main API server already had connections to this data store, and were serving requests okay. But over time as usage went up and down and demand for resources increased, more instances of this server were spun up, and the new instances were unable to talk to this data store, and would ultimately start serving errors.

Ultimately we tracked the issue down to this DNS issue and manually configured the required IP address until our hosting provider posted an all-clear on their issue, a number of hours later. After which point, we switched back to using the local DNS.