January 21, 2020

When you write a web server but you get served

Art: Unexpected Visitors, Ilya Repin, 1888

Served

If you go to Nginx.com, the company behind one of the world’s most popular web servers, you’ll find the most normcore SaaS website in the world. A standard product splash page with filled out with a list of product offerings. A happy customer pull-out quote singing its praises. And of course, the icons that show you’re a Serious Tech Company with arrows pointing heavenwards into The Cloud.

What you won’t see is what’s going on behind the curtain: the beginnings of Nginx as an open-source software project in Russia and its very recent sale to an American company. 

Perhaps the most important omission is that, in December, the founder of Nginx was detained by the Russian police, and both the Moscow offices of Nginx and his apartment were searched. 

There’s a whole lot of backstory left out, too: the involvement of Russian oligarchs, scores of lawyers, the CEO of the chief bank of Russia, and, of course, Putin Telegram stickers.

Let’s start at the beginning, though. 

How Websites Send You Stuff to Read

Let’s say you want to read a website. Perhaps you want to check out Hacker News to see if Softbank has closed up shop, leaving behind a trail of Saudi oil dollars, or read some fun social commentary. 

What happens behind the scenes between when you go to news.ycombinator.com?  

Allow me to illustrate with this bespoke late 2010s visual: 

Your local computer, via your browser, makes a request to the web server that’s hosting the website, saying “Please load this webpage for me.”

It does so using HTTP, a protocol, or language that computers use to talk to each other. Your machine tells the other machine, using a GET request, “Hey, send me to this website, and give me all the assets (images, text, cookies, whatever), that are available to me as an end consumer at news.ycombinator.com. “

In actuality, this all happens behind the scenes when you press the enter bar in your browser. 

But you can also run a GET request through the command line, using an application named cURL (which, like Nginx, was also started and maintained by a single dude.)  And when you do it that way, you get not only the content, but a bunch of interesting metadata about the site, also known as headers: 

Here is part of HN’s headers: 

curl -i -G https://news.ycombinator.com
HTTP/1.1 200 OK
Server: nginx
Date: Tue, 21 Jan 2020 11:26:26 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive

Julia’s wonderful zine covers a lot more of HTTP in very accessible detail if you’re interested. They’re a really fun rabbit hole to fall into (and, I’d argue important to understand if you use the web), but there is another reason I’m throwing all of this at you.

How does the Hacker News computer know what assets to give you, when to give them to you, and how to deal with requests when more than one person wants to view the Ferengi posts at the same time? 

A piece of software on the physical server, also called a web server, acts as a manager of all requests and responses and coordinates giving out assets to all the computers who ask for them. 

There are lots of different web servers, but the specific server that we’re talking about is called Nginx.  It was initially written by a dude named Igor Sysoev, in Russia. 

Sysoev and the Server

In an in depth (and Russian only 😞) interview with xaker.ru, Sysoev talks about his background, and the history of Nginx. He was born and raised in Kazakhstan in a military family. After finishing school, he returned to Kazakhstan, where he worked in an institute associated with the USSR Ministry of Geology. At the institute, he was exposed to Iskra computers, and he started to learn how to program. His first job out of college was as a sysadmin at a company related to trading oil products. He then went on to work at a site called XXL.RU (which is a surprisingly tame site in spite of its name, offering web hosting services). 

After XXL.RU, he went to Rambler, one of the early internet companies that popped up, mimicking America’s online ecosystem: 

As with the American internet, the Russian internet’s early days were full of whimsy, fun, experimentation, and lots and lots of piracy. Many of the things the early Russian internet did mimicked America, because that’s where all the innovation was. Rambler and Yandex, the first search engines, were based on Yahoo and MSN Search. Yandex still functions as today’s Google in Russia and has diversified into areas like taxis, traffic, food delivery, and much more. Vkontakte, a Facebook copycat site run by Pavel Durov (we’ll get to him later), was also founded around this time. 

Rambler is a search engine, launched in 1996, that also includes email, news aggregation, and other Google-like services. Interestingly enough, Google tried to buy a piece of Rambler in 2008, but the Russian government blocked the purchase under Russia’s antitrust act. 

As this was all happening in the business world, on the tech side, the most popular web server was Apache, which gets its name from the fact that the creators had to keep constantly patching it. A PATCHY web server slaps knee with laughter.  But because Apache, and other web servers at the time, had these issues that needed to be patched, Sysoev started contributing to Apache and working on those issues, when he was at XXL. 

In 2000, he started working at Rambler as a system administrator, and, in his own words, writing code on the side. The first thing he started to do was to contribute open-source code to Apache by...writing patches for modules. Eventually, he gained a ton of experience about how servers should work, got tired of writing patches, and decided it would be easier to write a web server from scratch.  

What he wrote became Nginx (pronounced as Engine-X. Interestingly enough, a lot of people in Russian pronounce it as INGEENX, or INJEENKS. )

Rambler’s photo service was only the third site to adopt Nginx. This fact is important, because it helps prove that he didn’t develop Nginx specifically for Rambler, but as an outside project.  After that, it became popular among Russian developers, and, then, eventually made its way into the English-language sphere. It’s Sysoev’s belief that this was because Nginx “just worked.” 

Along the way, Nginx added a lot more features and grew in popularity as a stable server able to serve tons and tons of traffic without so much as a blip. The Russian userbase were converts from the beginning. There are very charming threads on the Russian-language Nginx mailing list wishing Sysoev a very happy birthday. 

С Днём Рождения, Игорь!  Пусть озарят и согреют тёплые лучи сентябрьского уходящего солнца твою душу! С поздравлениями из Риги,

Happy Birthday, Igor! Let the warm rays of the passing September sun
warm your soul. With congratulations from Riga. 

Nginx Grows Up and Out 

Soon, because of the quality of the code and because it offered other features like load balancing, Nginx grew beyond a hobby, and started to turn into a very lucrative business. 

Today, Nginx is one of Russia’s biggest open-source success stories. It runs on more than 30% of the top million sites on the web, including high-traffic sites with LOTS of assets to serve, like Netflix, Dropbox, Pinterest, Airbnb, Soundcloud, and many, many more. 

And, if you look closely, you can see that Hacker News runs Nginx, too. Where’s that cURL command again? 

curl -i -G https://news.ycombinator.com
HTTP/1.1 200 OK
Server: nginx
Date: Tue, 21 Jan 2020 11:26:26 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive

Sysoev left Ramber and founded Nginx.com, the company, in 2011. (Nginx.org, the web server, remained free and open-source.) It called on investors and raised $3 million dollars from several Russian investors, as well as some in Silicon Valley: Runa Capital a Palo Alto firm with a focus on Russian companies, MSD Capital, run by Michael Dell (remember Michael Dell?!), and Aaron Levie, the CEO of Box. 

Then, in 2012, as is typical for many open-source projects that try to monetize, like cURl, and more recently, Dask, it started offering paid services in installing and maintaining Nginx. 

What’s always interesting to me about these stories is two things. One, that open-source software is usually not self-sustaining. People end up growing companies around it.

And second, and perhaps more important,  is how political software becomes. 

For example, as I’ve written before, the ties between the American and Russian internet are often much closer than the mainstream media would have you think, for example, how Americans had helped establish the Russian internet. 

Given how much flack Russia has caught in the American press over the past couple years, combined with the history of animosity between the two countries during the Cold War, you’d be forgiven for thinking that the countries’ internets have always been at cross-purposes, too. 

But the story of the Russian internet is very much intertwined with the history of the American internet, and I think this is really important to keep in mind as we read today’s American press, full of criticism of Russian hackers and bots.

Then, there was Google who tried to buy Rambler, but failed. 

Then, Elon Musk was chilling in Russia, trying to buy rocket parts.

And, finally Sysoev. Here is a dude initially working in Russia, on American-founded open-source projects, that impacted him to write his own open-source project, in Russia, which then grew to be international, and runs at big, American companies like Dropbox and Facebook. 

Servers for Sale

After several years as a company, Nginx successfully started raising money from even more high-rolling investors: 

NGINX had last raised money nine months ago, a $43 million round led by Goldman Sachs to fuel expansion, and had positioned itself as a strong alternative to F5 in recent years. (It had not disclosed its valuation in that round.) F5 itself, by coincidence, was said to have retained Goldman Sachs in 2016 to field acquisition interest in itself, although that never led to anything.

Finally, in 2019, after years of healthy growth, the company was acquired by an American company, to the tune of $670 million. F5 Networks, based in Seattle, now owns Nginx, and if you look on the website, it’s not obvious at all that it’s run and owned out of Moscow by the creator of one of the most significant web technologies of the past fifteen years.  Sysoev, the creator, isn’t even on the “About” page. (It redirects discreetly to F5’s leadership team.) 

Ruh-Roh Rambler

Of course, success is always bittersweet in Russia. That’s why we even have a saying, “Жизнь, бьет ключом. И все по голове.” The first part, “Life’s in full swing,” literally translates to, “Life beats with a key” and people usually add the second part, “And always over your head.” Basically it means that even when life is great,something is about to go very wrong. And it did. 

Rambler, the search company that Sysoev worked for all those long years ago, got super-jealous when the company was sold to F5. In late 2019, they figured out that they had unwittingly lost out on a ton of money by letting Nginx slip through their fingers.

So they, led by Alexander Mamut, a former lawyer, investor, and now head of the Rambler group,  sued Sysoev for developing Nginx under Rambler time. Here is how Mamut has been described: 

With several other businessmen, he played a key role in bankrolling Mr Yeltsin’s successful re-election campaign in 1996 and throughout the 1990s built up a series of holdings in the financial sector. 

He remains very close to the Kremlin and he is understood to be close to Russian prime minister Vladimir Putin, with one source describing him as “Putin’s man”. 

Speaking to the Daily Telegraph two years ago, Mr Mamut said he did not “see myself as an oligarch”, adding that he liked business where there was “an element of creativity”. “You can’t do business without money, of course, because that’s how it works but I’m not in business only for money”. 

Mamut incited a criminal case against Sysoev, claiming that Rambler owned the rights to Nginx and that the money from the sale should have gone partially to them. The result was the raids. 

Usually, the end of the year in Russia is a quiet time. While New Year’s, the most important holiday in the country, is fast approaching, work comes to a standstill. But this year, in late December, Maxim Konovalov, the co-founder of Nginx, opened his door early in the morning and found himself face to face with the OMON, Russia’s riot police/special forces. 

 Konovalov said to Russian Forbes,

Ко мне пришли в 7 утра, ОМОН с автоматчиками», — рассказал Коновалов. «Какие-то люди ходили с моей фотографией по подъезду и выясняли, где я живу, хотя я никогда не скрывался», — рассказал он.

[they] came to my apartment at 7 in the morning, OMON (Russia’s special forces/riot police) said Konovalov. “Some people were walking around with my picture in the building and found out where I live, although I’ve never hidden it, he said.

Sysoev’s home was also raided, and both men were detained by the police, and their phones were searched. Nginx’s offices in Moscow were also raided, according to some tweets by an Nginx employee that were later deleted, at the request of the Ministry of Internal Affairs, which controls the Russian police. 

Rambler’s Claims to Nginx and the Internet Under Attack

Sysoev, in the interview with xaker.ru, said specifically that contract he signed with Rambler when he started there stipulated that he worked on open-source stuff on the side. Moreover, writing code was not a part of his day-to-day duties as a sysadmin (oh how times have changed):

So he couldn’t have been working on stuff that Rambler could appropriate, unless they proved that he did that work on their servers, more than 15 years ago. Moreover, he started Nginx the company after he left Rambler. The case should be watertight, right?

In a just system, sure.  But as I’ve written before,  the business and political ecosystem behind the Russian internet is not a paragon of stability. For example, from the government side, the Russian government is already trying to turn the Internet off: 

The main problem with Russia’s internet is that the Russian government is trying to turn off the connection to the outside world. And, if it succeeds, other countries will want to follow its blueprint. 

…

This year, Russia cracked down on VPNs, which Russians have been using to circumvent the ever-increasing level of censorship. RKN approached VPNs and also asked them to ban the same websites that Russia does. VPNs, who are not under Russian jurisdiction, fought back. In 2018, the government tried to block access to Telegram, Russia’s favorite messaging platform, run by Pavel Durov. 

And here’s the attack on one of the fundamental backbones of the internet, web servers, from the business sphere. However, it’s not entirely the business sphere, if you look closer. Because Rambler is now invested in, at a 46% stake, by Sberbank, the largest bank in Russia. 

Sberbank also owns part of Rambler’s direct competitor Yandex, the Russian equivalent of Google: 

Rambler Group is a smaller rival to Russia’s most popular search engine Yandex, in which Sberbank holds a so-called golden share that gives it special veto power. 

What does Sberbank want with these properties? Well, Sberbank is tied directly to the Russian government, not only because it’s included in the constitution (the Russian central bank owns a 58% stake)  but because of German Gref. 

Who is German Gref? 

German Gref is the CEO of Sberbank. He worked in St. Petersburg during his early career, where he ran into V.V. Putin. Now, it could be that Gref was a fantastic lawyer and a hustler, but it’s remarkably interesting that his career started to really take off when he and Putin crossed paths, much as was the case for Mamut. 

He served in his government, and is now the CEO, where he’s taking on a digitalization program on behalf of Putin, based on the successes of whatever it was that he did at Sberbank (apps? Shorter wait times? Vodka-flavored lollipops for kids at Sberbank branches?)

And boom, here it is: 

This week it was announced that Sberbank has moved on very quickly: it has bought 20 percent of Mail.ru, one of Yandex’s major competitors. This means Sberbank is now not only Russia’s biggest bank, but will also become the number one player on the personal data market.

You might not want to share all your secrets with these men, but both Sberbank and Mail.ru own dozens of popular online-services and mobile apps that mean Sberbank now has access to the biggest stash of personal data in Russia.

Ok! So now, Mamut trying to appropriate Nginx for the purposes of Rambler, which is partially owned by Sberbank, which is partially owned by the Russian government. And both Mamut and Gref are directly connected to Putin. Is the picture starting to come together? 

Nginx now

Let’s loop back to where we are now with poor Nginx. As of January, the criminal case has still not been closed, even through Gref expressed disappointment with it, said he personally intervened, and said it should have been resolved in civil court. 

However, the criminal case is still not closed.  A meeting between Gref, Sysoev, and Konovalov, which was supposed to happen at the end of the year, didn’t.

There is still no mention of any of this on the Nginx website, or on Nginx.org, and there is a small mention on Sysoev’s own very draconian site, which reads, simply 

13.12.2019

Читаю новости, читаю Хабр и Роем.ру.
Всем большое, ОГРОМНОЕ СПАСИБО за поддержку!
Очень ценим.

    12.13.2019

    I’m reading the news, I’m reading Habr.ru [a tech website] and roem.ru.
    Thank you to everyone - a big, huge thank you for your support.
    We value it very much. 

The rest of the story was picked up by Russian news sites like The Bell, which was picking up the trail even as tweets were being deleted. 

So what? 

All of this brings to mind a number of discussion questions

What does this all mean, for Russian business, for the Russian internet, and most importantly, for the international internet? It’s not super great news that a company that was acquired by an American company is promptly facing an extremely unreasonable counter-attack from its Russian host, which it separated from years and years ago. 

What does it mean if your big American company company runs on open-source software that originated in Russia (or really any country with a political system that your country doesn’t agree with), where the government is now trying to hold the tech hostage? 

What does it mean for other creators of open-source software when they do work on employer time?

What does it all mean for Sysoev, who is now attracting the kind of attention he probably never wished for when he created Nginx?

For me, one of the main takeaways is this. Russian and American business connections are much, much more intertwined and complex than the mainstream media would have us think. And, just as importantly, software is not (or maybe never has been) a purely technical thing. It’s messy, and, at the end, there are always people and politics involved, even if it’s just a web server. 

What I’m reading:

All these lovely tweets from people who have bought Normcore stickers!

About the Newsletter

This newsletter is about issues in tech that I’m not seeing covered in the media or blogs and want to read about. It goes out once a week to free subscribers, and once more to paid subscribers. If you like it, forward it to friends!

Archive
Normcore Swag: Stickers. Mug. Notepad.

About the Author:
I’m a data scientist in Philadelphia. Most of my free time is spent wrangling a preschooler and a baby, reading, and writing bad tweets. I also have longer opinions on things. Find out more here or follow me on Twitter.