Good Rings don't scale
Art: Ring around the Rosy, Maurice Prendergast, 1903
The thing that sticks out the most about the Philadelphia office of Ring — the Amazon-owned digital doorbell company currently in hot water for a number of shady dealings — is not anything remotely related to security.
It’s the lifesize cardboard cutout of Shaq that stares at you with an aggressively optimistic smile. When I first spotted it, I felt his eyes follow me as I made my way through a series of interviews for a data role there several years ago. (I ended up not taking the position, but the experience stayed with me.)
Shaq-endorsed doorbell
Shaq is one of the celebrity endorsers of Ring. The story of how he got involved with Ring is pretty simple,
He recently bought a new house, and when he called a traditional security company looking for security cameras, they quoted him at $40,000. He wasn’t willing to pay that, so he bought a Ring doorbell camera for $300 and loved it. Then he sought out Ring’s CEO. “I said, listen. I love your product. If you need me to help you get it out there, let’s talk business.”
This isn’t surprising, seeing as to how Shaq endorses nearly everything:
If you watch sports on television, there is almost no avoiding Shaquille O’Neal. He’s an NBA talking head on TNT, but separately, he also pops up in advertisements for Carnival Cruise Line, Gold Bond, Icy Hot, Krispy Kreme, National General Insurance, and Ring home security, among others. He now makes more money each year in endorsements than he earned each year playing in the NBA.
(As a side note, I wish I could write a whole post just about Shaq’s hustle because it’s insanely impressive. He brings in $60 million a year in endorsements and business dealings. Shaq has recently upped his game, including joining the board of Papa John’s. There could also be an entire piece about how crazy Papa John has become after being spurned by the board, including eating an alarming amount of pizza in recent months.)
Despite some calls for him to step down as a spokesman from the likes of the Electronic Frontier Foundation (EFF), Shaq was at CES a couple days ago with Guy Fieri, wearing a Ring shirt and making cocktails.
Given that he’s such a savvy businessman, I was surprised he didn’t lay low. Ring has been on a pretty bad media streak recently.
In the latest round of headlines, hackers have hacked into a webcam that was pointed at a bedroom where girls were playing and talked to them directly, causing the CEO, Jamie Siminoff, “to cry”. (In retrospect, Siminoff has done a lot of crying in public interviews.) Leaving aside the very weighty implications of a culture where surveillance is so normal that a camera in a room for kids older than preschool raises no eyebrows, this is horrifying. There have been multiple other incidents, including an incident where a hacker was able to look into a bedroom, compromised login credentials, and video streams left entirely open to contractors through files available in AWS S3 buckets:
Despite its mission to keep people and their property secure, the company’s treatment of customer video feeds has been anything but, people familiar with the company’s practices told The Intercept. Beginning in 2016, according to one source, Ring provided its Ukraine-based research and development team virtually unfettered access to a folder on Amazon’s S3 cloud storage service that contained every video created by every Ring camera around the world. This would amount to an enormous list of highly sensitive files that could be easily browsed and viewed.
...
At the time the Ukrainian access was provided, the video files were left unencrypted, the source said, because of Ring leadership’s “sense that encryption would make the company less valuable,” owing to the expense of implementing encryption and lost revenue opportunities due to restricted access.
Then, there is the thorny issue of collaboration with the police to spy on people - and then Ring turning around to spy on the police.
That’s a lot of problems. How did Ring get to this point?
Ringception
The company started as the brainchild of Siminoff, who was a mediocre student but enjoyed hustling and building things. He studied entrepreneurship from Babson College and made his first money by writing business plans for other college kids.
After he graduated, he went back to creating and went through various loops of building things,
While spending long hours in his work space, he became frustrated that he was unable to answer the doorbell without being interrupted. Siminoff began looking for a doorbell that rang to his phone and was surprised to find that there was nothing like it on the market.
“I was like, how the fuck can there not be a doorbell that goes to your phone?” Siminoff told Digital Trends.
To bring his prototype to market, Siminoff needed to compete, and he landed on the TV show Shark Tank, where VCs snap-judge you as ominous music plays in the background. The sharks passed on him, but he got the marketing boost he needed and soon attracted the attention of Richard Branson and Shaq.
Zonoff
Around the same time, a company called Zonoff was percolating in Philadelphia. I heard about them through the local tech news. They were doing something cool: trying to connect all the devices in the home. This was before the Internet of Things (and @InternetofShit), and no one knew or thought about the hackability of these devices. Zonoff kept growing and growing, and soon attracted the attention of Honeywell—the traditional thermostat company— to the tune of $40 million.
The sale fell through, and Ring swooped in, hired all of the Zonoff staff as full-time employees and created a new team, Ring Solutions, that focused on the holistic home IoT strategy.
That’s when all hell broke loose: ADT was also dragged in. As a previous investor, at $31 million, ADT claimed that it (Zonoff) had stolen ADT technology,
In a lawsuit filed in the Delaware Court of Chancery, ADT alleges Harris stole the intellectual property that rightfully belonged to ADT as a secured lender to Zonoff.
Law360.com reports that ADT was Zonoff’s largest investor (there had been speculation), and that the security and home-automation giant had made equity investments as well as “large, secured loans” to Zonoff beginning in 2014.
CE Pro broke the news last month that ADT might have provided debt financing to Zonoff, in which case: “If, as likely, Zonoff files for bankruptcy, ADT might come to own its assets.”
But the lawsuit madness eventually subsided, and then Amazon purchased Ring for a billion dollars, one of its biggest purchases ever. .
Ring made bank. Amazon got into the IOT game for real. Jamie bought a mountain bike. Everything was going great. Then the exposes started coming.
Complications
As I read the headlines now, I try to recall anything about my interview experience that struck me as excessively odd, or indicative of any of their recent transgressions and couldn’t think of anything.
When I was taken around the office, everyone was busy working on extremely normal tech company things.. There were teams doing security, moving data around in AWS buckets, streaming things, creating dashboards. My interviews were also nothing out of the ordinary, including some time spent discussing the various pain points of AWS architectures.
But as I watched people busy writing code, taking customer service calls, and monitoring operations, I started to have some doubts. When one of my interviewers mentioned that if I went on to work at Ring I’d also get a free doorbell, as a perk, I became uneasy. I realized, then, that I didn’t want an internet doorbell. I didn’t want to have my house connected to the Internet of Things, particularly if my coworkers could see the logs and could access the video.
But no one said anything about this big philosophical dilemma hanging over the whole operation: what happens when machines start watching neighborhoods instead of humans?
What happens when surveillance and fear become the norm?
But gradually, Ring’s hold on my brain deepened. And it started to seem more sinister. The thing has a motion sensor. Whenever something trips it, the phrase “There is motion at your Front Door” dutifully pings your phone. And goodness, look at that language. “There is motion” is the kind of military-industrial abstract anxiety that DeLillo’s made a career off of. Even the capitalization of Front Door adds an imperial twist: Enemies at the gate! Valar morghulis!
Then last fall our son descended into a season of poor health and repeated hospitalizations. We were up nights when he needed treatment, and we were up on other nights standing a jittery watch. During those nights, I saw how many times Ring sent that “motion at Front Door” alert to my phone. I’d rise at 1:15 thinking I heard a cough through the baby monitor and would see that the “motion” alert had been triggered a half-dozen times. Why? How? Though I knew in my core that all was fine outside, I still felt obsessed.
And when we then normalize unmonitored cameras taping people and putting them on display for laughs without their consent?
What about when a private company becomes a defacto police force?
Systems are still only people, but harder
Ultimately, Ring is nothing more complicated than just people watching you. Only the people are watching the system that watches you instead. In a home run of a blog post, Adrian Colyer’s Morning Paper covered a paper from 1983 called “The Ironies of Automation.”
The gist of the paper is that, as machines become more complicated, the more skilled people have to be in understanding the system so that they’re ready to intervene in the system when things inevitably go wrong
In an automated system, two roles are left to humans: monitoring that the automated system is operating correctly, and taking over control if it isn’t. An operator that doesn’t routinely operate the system will have atrophied skills if ever called on to take over.
Not only are the operator’s skills declining, but the situations when the operator will be called upon are by their very nature the most demanding ones where something is deemed to be going wrong. Thus what we really need in such a situation is a more, not a lesser skilled operator!
There are a whole bunch of things to do to mitigate this, but all of this is compounded by the fact that an automatic doorbell is not one system - it’s multiple ones. There are no online architecture diagrams for Ring that I could find (thankfully!) but here’s a reference architecture, a simpler setup that walks you through what it might potentially take to establish a smart doorbell in the cloud.
Look at this! Keys, Message APIs, schedulers, Service Buses (you hate to see it), SQL databases, and the physical hardware itself. And this is just for one doorbell! Imagine if you’re serving purportedly millions of households, as Ring is. And if you don’t have just a doorbell, but a smart home. Each of those APIs, every single thing you connect to the internet is an attack vector. Even more Normcore than that, each of those things has the potential to fail, as happened with the connected Nest thermostat, plunging households with babies to very cold temperatures overnight.
And what about the people side? How do you account for the fact that this system normalizes fear and surveillance across the country? And that, as a company, you’re now collecting extremely sensitive, liable, and fragile data. The more employees that have access to this very, very, very sensitive data the more potential there is that it gets leaked, looked at by the wrong party, or hacked. It doesn’t matter how good your company is.
And, sure enough, here we are:
"We are aware of incidents discussed below where employees violated our policies," the letter from Ring, dated January 6, reads. "Over the last four years, Ring has received four complaints or inquiries regarding a team member's access to Ring video data," it continues. Ring explains that although each of these people were authorized to view video data, their attempted access went beyond what they needed to access for their job.
"In each instance, once Ring was made aware of the alleged conduct, Ring promptly investigated the incident, and after determining that the individual violated company policy, terminated the individual," the letter adds. As well as firing workers, Ring has also taken steps to limit such data access to a smaller number of people, the letter reads. It says three employees can currently access stored customer videos.
In my interviews, no one talked about any this, and, of course, there was no way I could bring it up. At most companies, people don’t talk about stuff like this on a day-to-day basis.
This is not because people are evil and they want hackers to get into your webcam. There are not Good People and Bad People. There are no evil villains in underground lairs trying to run companies (except for Adam Neumann and Larry Ellison.) The world is more nuanced and multidimensional than that.
There are, at any given company (yes, including Facebook) for the most part, just people trying to do their jobs as best as they can given what they know with their limited point of view, incentivized by the pressures exerted by the market on their companies.
Ultimately, one of the main problems is that these kinds of large, complex systems (like, for example, social media networks, or the American economy) are enormously hard to steer and predict the outcomes for.
How could Jamie have known, back in his garage, that some random would hack into a webcam and terrify two little girls? Human brains are not designed to think through sets of conesquences in this many dimensions.
Good Rings Don’t Scale
And, ultimately for Ring, this is where the problem is.
Siminoff started the company in his garage, as a personal hack solution to his own niche use case. And then, for years afterwards, here he was, building a company, scraping money together, pitching, and, again, fighting lawsuits. He had to make money, and then, later with Zonoffo, manage a second half of the company all the way across the country. (He also had to make time to cry in interviews), and trying to make this very impossible thing work.
He didn’t think through all of the possible outcomes that could happen, because it’s impossible to think that this hacked-together doorbell that works for your garage will contribute to the rise of surveillance culture. He was busy just trying to survive.
And so, as the company grew, Siminoff focused on the successes and the leading indicators: the Shark Tank reviews, the media circuit, the sale to Amazon, the cardboard cutouts of Shaq.
And, in that capacity, in the sales culture, he succeeded beyond his wildest dreams. He cashed his check and won’t have to think about the implications again. He can even leave the company if he wants to.
But the thing he created is now much, much bigger and more complicated than him, and now involves millions of homeowners, a budding culture of it being ok to record anything and anyone near your doorstep without permission, the normalization of IoT in the home, police departments, and other social and political implications that will take years and years for society to untangle. Good things don’t scale, and neither does the local neighborhood watch.
So, on second thought, maybe Shaq was right to go to that cocktail mixing session, and maybe I’ll take a screwdriver. Or five.
What I’m reading lately:
The latest issue of the O’Reilly programming newsletter (which is impossible to link to on an issue-by-issue basis), about where are we headed in 2020 for 6 different programming languages
This excellent read on data science consulting by Ethan. Rings true with my experience.
Listening to this interview with Josh on his time at Slack. I am not exaggerating when I say it’s one of the most interesting podcasts I’ve listened to lately that’s fundamentally changed my understanding of Google and FB from a software architecture perspective.
A very Normcore dystopia
Learning your own language:
About the Newsletter
This newsletter is about issues in tech that I’m not seeing covered in the media or blogs and want to read about. It goes out once a week to free subscribers, and once more to paid subscribers. If you like it, forward it to friends!
Archive
Normcore Swag: Stickers. Mug. Notepad.
About the Author:
I’m a data scientist in Philadelphia. Most of my free time is spent wrangling a preschooler and a baby, reading, and writing bad tweets. I also have longer opinions on things. Find out more here or follow me on Twitter.