Almost one whole sleepless week: the aka-aki development team managing the crisis

What a hard week this has been! For the team here at the aka-aki offices in Berlin and of course for you, our users! Servers broke down again and again, errors in friends lists, those mistyrious zombie aki mails that appeared again and again after being deleted and loads of other annoying stuff. About what went wrong exactly, we talked with our Head of Web Development, Lukas. Here are his answers…

Lukas, the problems already started last week. What went wrong?

Last wednesday our iPhone app had been approved to the iTunes store. It’s not an excuse, but after being rejected again and again for almost three months, this kind of hit us by surprise. And what we really dind’t expect was the enormous success: within two days aka-aki became the most popular social network app in the German app store and climbed to rank 4 of all free apps. The flood of new users was too much for our server. Since then we added one additional server after the other. But there were more problems with migrating databases etc. The system’s behaviour remained weird and unfortunately there were break downs again. We had to learn, that you just cannot simulate everything in advance. And that we are still a small company with limited ressources. Concerning man power and technical equipment we still cannot compare ourselves with the “big ones” in the internet industry. The only thing we can do now is to apologize to our users for the problems. We are really sorry.

And at the moment? How is the system working right now, is it stable again?

Things are going better now. We haven’t expearienced any major problems over the last hours. From time to time there are short interruptions in the service that are caused by hardware problems that popped up last night. So the system still keeps us busy. But the users can relax. If there are still problems with logging in please try again a minute later and verything should work well.

What have you and your team changed so far to avoid those problems in the future?

Currently we have 10 servers working where there were two before. We added capacities in a second data center, which means more reliability. Thanks to the folks at filoo GmbH for the quick help. You did a great job! We also enhanced the monitoring so that we can react more quickly when something goes wrong and the system shows weird behaviour. And we will change the iPhone app to reduce the load of requests to the server.

Lukas, thank you for your answers. We wish you and your team a quiet night and a good sleep. You deserve it.

12 Responses to “Interview: What went wrong last week?”