Robotic vacuum cleaners couldn’t be summoned. Whole Foods orders were suddenly canceled. Parts of Amazon’s mammoth retail operation slowed to a standstill.
Amazon Web Services, the leading provider of cloud infrastructure technology for businesses large and small, was hit with a historic, hours-long outage on Tuesday. Popular websites and heavily-used services were knocked offline, angering users and underscoring the severity of problems that can arise from having so much economic activity reliant on technology from just a few vendors.
AWS controlled 33% of the global cloud infrastructure market in the second quarter, according to Synergy Research Group, followed by Microsoft at 20% and Google at 10%. Revenue at AWS jumped 39% in the third quarter from a year earlier to $16.1 billion, outpacing growth of 15% across all of Amazon.
Tuesday’s outage began around 11 a.m. ET and was mostly resolved by Tuesday night. Amazon confirmed that service issues with AWS’ main US-East-1 region, located in Northern Virginia, were causing problems for its warehouse and delivery network. The company hasn’t said what caused the outage.
Fulfillment center and delivery operations were brought to a standstill in some pockets of the U.S., Amazon said. The outage took down internal apps used to scan packages and load delivery routes, according to workers’ posts in Facebook groups and a notice sent to drivers that was viewed by CNBC.
Workers were told to stand by in break rooms and loading areas, and an Amazon driver tweeted a video of a co-worker performing karaoke in a warehouse.
Whole Foods, which Amazon acquired for $13.7 billion in 2017, canceled orders for some users in affected areas, offering refunds as a consolation. Amazon Flex drivers, contractors who make deliveries using their own vehicles, were promised pay after being sent home because shifts were unavailable, according to a notice from Amazon.
The AWS snafu crippled Amazon’s retail operations at a particularly inconvenient time. The company is in the middle of peak season, when it’s hit with a flurry of orders from holiday shoppers. Third-party merchants, who make up more than half of all retail volume sold on Amazon, rely on a few weeks at the end of the year for an outsized percentage of their annual sales.
Joe Stefani, an Amazon seller in Chicago, said his business, Desert Cactus, couldn’t get inventory into the company’s warehouses due to the outage. Stefani said Amazon handles 90% of his company’s orders, shipping products to customers from its fulfillment centers.
Sellers like Stefani weren’t able to access Seller Central, an internal system Amazon uses to manage customer orders. That meant Stefani was unable to print out shipping labels that are required for any shipments sent to Amazon warehouses.
“We could not send in at least 10,000 to 12,000 items,” ranging from NBA lanyards to NHL merchandise, Stefani said. “It will end up costing us money in the long run.”
Other major web services and infrastructure companies have seen significant outages this year. Fastly, whose technology helps companies speed the delivery of digital content to consumers, experienced an outage in June that took down major websites like Amazon, the New York Times and Hulu. In October, Facebook suffered its worst outage since 2008 due to a configuration issue.
Amazon has had its own disruptions in the recent past. AWS experienced an outage in November 2020, when problems with a service called Kinesis brought down a host of websites. This time the damage was more widespread, affecting businesses of all shapes and sizes.
Roombas, smart cat litter boxes taken offline
Evan Coleman knew something was wrong when he couldn’t load an app connected to his self-cleaning cat litter box, which had just arrived in the mail.
Coleman, lead software engineer for wedding planner service The Knot, logged onto Twitter and saw a flurry of posts about an AWS outage. He then noticed that his app-controlled ceiling fan and web-connected cat feeder weren’t working either.
Thankfully for Coleman and his cats, Leo and Luna, the feeder could still dispense food manually.
“The cats did not starve,” he said.
Steve Peters, an experience designer in California, had a different kind of mess on his hands. After eating his breakfast Tuesday morning, Peters summoned his iRobot Roomba vacuum to clean up his crumbs.
The Roomba app relies on AWS.
“I was forced to dig in the closet and find a dust pan for crying out loud,” Peters said. “I’m just glad I don’t have an internet-connected fridge.”
The outage also took down Canvas, an online teaching platform with more than 30 million users, as well asthe LockDown Browser from Respondus, a test proctoring service that blocks certain web browsing functions while students are taking an exam.
The timing was as bad for educational institutions as it was for retail, since many college students across the country are in the middle of finals week. Chantal Lamourelle, an assistant professor at Santa Ana College in California, said she had to delay an exam for her childhood development class that was supposed to be taken online Tuesday morning.
Ananay Arora, a senior computer science student at Arizona State University, said he couldn’t access materials to study for an upcoming exam. On Tuesday, he received a flurry of Discord and Snapchat messages from friends at other universities who were experiencing similar issues.
While outages like Tuesday’s feel catastrophic in the moment and can have significant ramifications on businesses and organizations that rely on the service, such disruptions are a rarity, said Carl Malamud, a technologist and internet archivist based in the Bay Area.
It took Amazon roughly nine hours to resolve the issues — an impressive feat given AWS’ size and scale, Malamud said, adding that AWS and other major cloud services are generally very reliable.
“I bet they had all hands on deck,” Malamud said.