Tech At Wooga: Elli
My name is Knut, and I want to show you something really cool that I built to solve some problems we are facing here at Wooga.
Having several very successful social games means we have a large number of users. In a single game, they can generate around ten thousand HTTP requests per second to our backend systems. Building and operating the software required to service these games is a big challenge that sometimes requires creative solutions.
As developers at Wooga, we are responsible for the user experience. We want to make our games not only fun and enjoyable but accessible at all times. To do this we need to understand and control the software and hardware we rely on. When we see an area where we can improve the user experience, we go for it. Sometimes this means taking on ambitious projects. An example of this is Elli, a webserver which has become one of the key building blocks of our successful backends.
Having used many of the big Erlang webservers in production with great success, we still found ourselves thinking of how we could improve. We want a simple and robust core with no errors or edge cases causing problems. We need to measure the performance to help us optimize our network and user code. Most importantly, we need high performance and low CPU usage so our servers can spend their resources running our games.
Having read the source code of Misultin, Cowboy, Mochiweb and parts of Yaws, I was unsure if it was even possible to achieve my first goal: a simple core that can be measured and easily readable while still implementing a complete webserver. If this was possible, I wanted to give a go at creating a webserver tailored to our high-volume needs.
In a Friday afternoon hacking session I had a very rough first version of Elli that actually worked, using the acceptor-becomes-request-handler idea found in Mochiweb. Encouraged by good progress, I hacked further. After a few rewrites, I had a very solid and robust core solving my exact problem and nothing more. To test if Elli would hold up in the real world, I put it to use in a service used by all of our games. It handles some hundred requests per second and has been running perfectly for six months. I’m very satisfied with the performance and simplicity offered by Elli.
Elli has directly helped us improve the user experience through less errors, higher availability and faster page load times. We finally have a tool that solves our problems very well and can handle the stress we put on it. Most importantly, using Elli allows us to focus our time on improving our games.
We find that creating our own software to solve very hard problems in a very specific way while completely removing all unrelated problems in the existing software saves us time. This works very well for specific problems, but not for general problems. While we happily create our own webserver, database driver and so on, we will not create our own operating system, programming language or TCP/IP implementation.
By giving away our software under a very free license, users will be able to make it better which directly benefits us, just as we improve the open source software we use. As a developer at Wooga, I’m actively encouraged to release open source software. It is even part of our performance evaluation.
Elli started out as an experiment and turned out to be a valuable tool in our toolbox. I hope it will also be valuable to others building robust high-volume, low-latency server applications. The project is available under the MIT license at github.com/knutin/elli.
Please check it out. I’m very interested in feedback and patches to improve Elli further.