On Monday afternoon, Amazon employees around the world were dealing with a nightmare — a massive AWS outage that made their work environment completely unstable.
Apparently, the cause of the problem was a host of different factors. An internet security firm warned that an outage at a third-party cloud storage server was the first indication that something was wrong with AWS, reported the Register. The firm, Crycen, also noted that they detected a serious potential issue with web browsing earlier in the day.
AWS did manage to secure the issue by switching their DNS services to their own servers before it could spread to its cloud storage services. According to reports, there was a bit of a scrambling of plans to make up for the outage before the technical team could gain control of the situation.
A spokesperson for Amazon told Forbes that “Customer service support for US-East West, US-East and US-East East West were restored at 4:39 PM PST Monday afternoon and all systems were fully operational within an hour,” later adding that they have “invested heavily in our customer support infrastructure and will continue to improve our training, communications and operations to ensure a stable service.”
Tuesday was far from easy for the staff at Amazon warehouses. Working conditions — and the workers themselves — are notoriously tough at Amazon warehouses, where the cloud-based systems, which allow their inventory to stay updated for the customer, are at the heart of the operation. Amazon is one of the world’s largest retailers and enjoys an economic perch that even some international companies cannot match. Amazon employees have to deal with hours-long shift work, angry customers and overcrowded, sardine-like areas with coworker strollers. And now, these vast warehouses are experiencing short working days.
A company spokesperson told the New York Times that workers “will receive time off for Monday,” and offered to place a business-as-usual message on the sign-in desks in their warehouses.
“There will be some employees who won’t be able to work because of the outage,” he added. “We will try to notify them as early as possible when we can. If employees have any questions, they should call customer support.”
UPDATE: Amazon says its main US-East 1/2 and US-East 2/3 warehouses are back up, and the problem is under control.
Amazon’s Americas HQ Amazon’s biggest warehouse in operation has also been impacted by Amazon’s biggest infrastructure failure in 18 years, an outage that will have no immediate impact on customers. The outage was triggered by failed storage drives of Amazon Web Services, which power AWS services like Simple Storage Service (S3), Elastic Block Store (EBS) and Crashplan. Late Monday afternoon, AWS found out that an additional 18 percent of their Big Blob [sic] servers – a central point for all infrastructure and storage that runs on the Big Blob – were down. This was an unexpected component of the failure that triggered the entire episode. AWS said in a statement that “100 percent of that service is now back.”
Read the full story at The Verge.
Related
Blockchain provided small German software firm with access to Amazon’s huge cloud
A space agency joins Amazon, Capital One to test blockchain-based credit card and financial system
Amazon’s cloud services suffer a major outage that has started to ripple around the internet