How Uber is working to prevent another Halloween outage

Oct. 28, 2018 / 10:12 PM GMT

By Alyssa Newcomb

For anyone who took an Uber ride on Halloween night in 2014, the scariest memory of the holiday might be waking up to a bill for hundreds of dollars for a quick ride home from a party.

Uber was of course a smaller company in 2014, and engineers weren’t prepared for the level of demand that ultimately brought down its supply and demand system during peak Halloween party hours, driving up prices.

The Halloween outage notoriously left one woman to crowdfund her $362 Uber bill, so she could afford to pay her rent on Nov. 1.

Fast forward to 2018, and on the heels of an upcoming IPO that could value Uber at $120 billion, the company has a site reliability engineering team with more than 100 people — and even more partners around the company — who conduct drills throughout the year to make sure they’re ready to handle any technology issues that could impact drivers and customers.

“When the platform is down, we measure that as not only a loss to the business in terms of revenue, but also to our partners [drivers],” said Donald Sumbry, a senior engineer at Uber who leads the site reliability team.

It’s difficult to quantify the cost of an outage, since they can vary in time and by location, the company said. Uber’s last two widespread outages, where customers were not able to order rides, happened in 2016.

In the run-up to Halloween festivities, Uber’s teams around the world have been working together to make sure nothing breaks.

“We start projecting out what we think our peak capacity is going to be,” he said. “We also want to know the volume of the capacity, so at any one moment, how many trips do we need to be able to support through all of our infrastructure?”

Uber’s engineers then run drills on the same infrastructure that serves real-life trips, adding a load of additional “fake” trips internally to stimulate how mission control will look on the big night.

“We’ve got this team of engineers basically sitting in a room and over the course of a few hours, we’ll keep adding load and monitoring all of our systems to ensure that we’ll always have a projection in mind, so maybe it’s number X,” Sumbry said. “We’ll just keep adding load until we get up to that number, and then sometimes depending on how it went, we’ll actually go a little bit past it until we either see something break or we see some stress on the system.”

Human eyes are always watching for any hints that could spell trouble, but the team also relies on sensors distributed throughout its infrastructure to add an extra layer of security.

“We could, for example, lose an entire data center and we could do that without affecting customers because our backing systems and infrastructure will actually just redirect current request and trips to other data centers sort of seamlessly,” Sumbry said. “I think that’s kind of the goal. We don’t want these big things to affect our partners and our users at all.”

Once Uber gets through Halloween without a scare, the company is set for a blockbuster fourth quarter. Halloween and New Year’s Eve are typically the busiest days of the year, but bizarrely the day predicted to be the craziest for Uber this year is Dec. 14.

The reason is anyone’s guess.

Uber tends to get peaks on Fridays and Saturdays, and those days are only projected to get busier during the fourth quarter.

“I think the reason why it drops after that is because people start traveling and going to their families and not commuting and not working,” Sumbry said. “And then we see it again on New Year’s when everybody goes out.”