Post-mortem on the data loss and server outage on August 9, 2023

On August 9, 2023, Capacities was suddenly unavailable for all users. The outage lasted four hours, and we had to restore all data from a backup. With this backup, we all lost our work from the 11 hours before the outage. We again apologize sincerely for this. It was a very rough day for us as well.

We are very thankful for all the kind words and support we received. Thank you all for supporting us on that day.

Since then, we have been working full steam on finding the root cause and on improving our infrastructure and security. Here are some insights:

• Root cause found: We know what caused the issue and resolved it so it cannot happen again. A very unlikely user action combined with a bug in our database version caused the corruption of our main database data. There was no hardware failure and no attack on Capacities.
• Shorter backup cycles: We reduced our backup cycles to three hours at maximum. We hope to never need a backup again, but at least the loss will be much smaller in the case of a very severe problem.
• More reliable and faster infrastructure: In the four hours of the outage, we moved all our services to a bigger infrastructure, allowing us to scale better from now on. Since the outage, we did not have any problems anymore.
• Better security: We worked with security experts and improved the overall security and infrastructure of Capacities. We are in the process of transforming Capacities from a beta version into a reliable and very secure product.

We continue to work on the foundation of Capacities. Many improvements in the next weeks will be under the hood and invisible to you. This does not mean that we're not full steam working on a better Capacities.

On top, we were working on better offline support for Capacities. We were hesitant to move the "offline support" ticket to "planned" because people have very different ideas about what "offline support" means. We'll improve more and more features so that they also work without an internet connection. We'll also store more data on your computer so you can access it when you're on the go. Nevertheless, Capacities will not become a local-first software in the foreseeable future. We wrote about this in detail here. We hope that makes sense to you.

If you have any questions, feel free to start a discussion in our community.

Again, we apologize for the data loss and thank you for your support.

Michael & Steffen