The Internet of Everything
The downside of our increasing connectivity.
The Atlantic’s Will Gottsegen points out “The Internet Is Going to Break Again.”
Everything is in “the cloud” now, except the cloud is a real place, and it’s in Northern Virginia. Rows and rows of servers stacked in Amazon-owned warehouses across Ashburn, Haymarket, McNair, Manassas, and Sterling make up a chunk of the infrastructure for the modern internet—equipment as crucial as railway tracks and the electric grid. When a technical issue disrupted operations at those facilities yesterday, it was enough to temporarily crash the internet for users around the world.
The incident marked at least the third time in the past five years that Amazon Web Services’ Northern Virginia facilities contributed to a widespread internet outage. This time, more than 1,000 sites and services were affected, according to Downdetector, costing companies an estimated hundreds of billions of dollars. Venmo users were locked out of their payments, and international banks experienced major blips in their service. People struggled to book urgent doctor appointments and couldn’t access their Medicare benefits. Snapchat and Reddit were down, as were Instagram and Hulu. Ring paused their doorbell cameras; ChatGPT stopped answering.
WaPo’s Leo Sands reports on one microeffect: “Smart beds flipped out during the AWS outage, and so did their sleepy owners.”
[P]erhaps the rudest awakening was felt by those sleeping on Eight Sleep’s high-spec internet-enabled mattresses. As the outage caused cloud servers to fail, people reported being awoken in discomfort by beds that locked in an upright incline, became unbearably warm, blinked flashing lights or even sounded a wake-up alarm.
“That is not the experience we want to provide and I want to apologize for it,” said Matteo Franceschetti, the chief executive of Eight Sleep, which also manufactures and sells internet-enabled mattress covers and pillow covers. In a social media post, he explained that the malfunctions were the result of the AWS outage and said engineers were racing to build an outage-proof mode in the event of a future outage.
Eight Sleep did not immediately respond to a request for further comment early Wednesday. As of Monday evening, Franceschetti said all devices were functioning although some were still experiencing “data processing delays.”
The company’s internet-enabled mattresses allow users to adjust their bed’s temperature between 55 and 110 degrees, elevate their body into different positions, as well as activate immersive “soundscapes” and vibrational alarms. The most advanced product bundles retail for over $5,000, in addition to a yearly subscription costing $199 to $399 that is required to enable the temperature controls.
The malfunctions also highlighted some of the risks in making so many aspects of everyday life dependent on cloud-based technologies, which Alan Woodward, a professor of computing at England’s University of Surrey, said will always be prone to occasional failure.
“Complexity always causes issues,” he said in a phone interview Wednesday. “The more complex you make the functionality, the more difficult it is to understand what will happen when some parts of that whole chain of computing fails.”
Gottsegen’s colleague Andrew Moseman laments a related issue in “My Car Is Becoming a Brick.”
For most of its short life, my Tesla Model 3 has aged beautifully. Since I bought the car, in 2019, it has received a number of new features simply by updating its software. My navigation system no longer just directs me to EV chargers along my route—it also shows me, in real time, how many plugs are free. With the push of a button, I can activate “Car Wash Mode,” and the Tesla will put itself in neutral and disable the windshield wipers. Some updates are more helpful than others: Thanks to Elon Musk and his middle-school humor, I can now play an updated array of fart sounds when an unsuspecting passenger sits down.
But Musk is already starting to leave my car behind. In July, Tesla rolled out a version of Musk’s AI assistant, Grok, to its vehicles. Even as a chatbot skeptic, I could see the usefulness of asking my car for information without having to fumble with my phone. Alas, at present Grok runs only on Teslas made in the past few years, which have a more advanced processor to power their infotainment system. My sedan is simply too old.
Cars used to be entirely mechanical objects. With hard work and expertise, basically any old vehicle could be restored and operated: On YouTube, you can watch a man drive a 1931 Alvis to McDonald’s. But the car itself was stuck in time. If the automaker added a feature to the following year’s model, you just didn’t get it. Things have changed. My Model 3 has few dials or buttons; nearly every feature is routed through the giant central touch screen. It’s not just Tesla: Many new cars—and especially electric cars—are now stuffed with software, receiving over-the-air updates to fix bugs, tweak performance, or add new functionality.
In other words, your car is a lot like an iPhone (so much so that in the auto industry, describing EVs as “smartphones on wheels” has become a go-to cliché.) This has plenty of advantages—the improved navigation, the fart noises—but it also means that your car may become worse because the software is outdated, not because the parts break. Even top-of-the-line phones are destined to become obsolete—still able to perform the basic functions like phone calls and texts, but stuck with an old operating system and failing apps. The same struggle is now coming for cars.
Software-dependent cars are still new enough that it’s unclear how they will age. “It’s becoming the ethos of the industry that everyone’s promising a continually evolving car, and we don’t yet know how they’re going to pull that off,” Sean Tucker, a senior editor at Kelley Blue Book, told me. “Cars last longer than technology does.” The problem with cars as smartphones on wheels is that these two machines live and die on very different timescales. Many Americans trade in their phone every year and less than 30 percent keep an iPhone for longer than three years, but the average car on the road is nearly 13 years old.
There’s no obvious solution for any of this.
That so much of the Internet—and, thus, modern society—runs on cloud servers provided by a handful of providers is a significant vulnerability. But diversification has its own issues.
That the so-called Internet of Everything has gradually spread brings both tremendous convenience and vulnerability. Most of us get our entertainment from streaming services. If they or our Internet connection go down, we don’t have a backup. That’s only mildly inconvenient and more than made up for by an array of choices that would have been almost unimaginable not long ago. But many of us now have doorbells, thermostats, and other central parts of the home networked. Some even have their refrigerators connected.
Moseman points to a related problem: as more of our everyday technology becomes reliant on connectivity, it becomes obsolescent faster. My Mazda CX-9 is relatively old. Aside from a navigation system that I usually bypass in favor of my phone’s, there’s nothing to “brick.” But it’s bizarre that a six-year-old car can’t run the manufacturer’s software.
Less crucially, purchases like smart TVs can quickly become outdated, if not worse. Apps that the customer bought the device expecting to run—they’re pictured right there on the box!—can become unsupported almost immediately. Several major manufacturers relied on Yahoo Smart TV to support various apps; they stopped working last month for older models.
I was 5 when we got our first family PC in 1990, 12 when we got the internet, 15 when we got a dedicated line so the internet could be ever present, and 22 when I got my first smartphone. I’ve been immersed in whatever the cutting edge tech has been for most of my life. In the last 5 years, though, I’ve found myself going backwards. Our new fridge with a touchscreen that came with the house died last year, replaced it with the most no-frills fridge I could find. Finally got a car built in this millennium, and spent extra time looking for a model that didn’t have onstar, wifi, or anything that demanded a connection. The Alexa we got as a gift 3 years ago has never been taken out of the box. My phone goes on a charger stand when I get home, and my office has just had to accept that if they need anything from me after 4:30 pm it had better be important, and they had better call. Being connected to everything at all times has just become too much of a hassle.
I get the feeling that there are many in my generation–or perhaps just my micro generation of vanguard millennials–who are moving in the same direction, but I think we are the only ones. Boomers have certainly embraced IoT, and GenZ and GenAlpha have never known a life without this type of connectivity. And I don’t know how long my efforts can last. Sure, I could find a luddite vehicle this time around, but in 10 years it will be impossible. My no-emails-after-5 scheme works in an office where I’ve established my worth for more than a decade, but the moment I work somewhere else I’m going to have to join the always-connected culture.
As a kid, we were just one of the millions of families that experienced the 1965 northeast blackout, so the internet isn’t the only thing that can cause system wide failures. But yes, we are increasingly vulnerable. Redundancy can reduce the risk, but that comes at a huge cost.
Oh, the blackout was and adventure.
How was the product developed, made, and sold without an outage proof mode to begin with?
Sure, AWS outages are not common, but the home internet going down, even for a few minutes, is common.
I don’t like the idea of connected gadgets much. Especially if they won’t work without an internet connection. I wouldn’t mind, say, a kitchen appliance that could relay its status to an app, so I could monitor without having to walk the few meters to the kitchen and look. But I wouldn’t spend any extra money for that functionality.
@Sleeping Dog: For sure. But, of course, if the power goes out, so does the Internet. We have a Generac that runs on propane that partially backs up our power, including our Internet router.
True, but not unprecedented. To this day, cable cuts out now and then.
That’s worse, but not hopeless. Before I had a “smart” TV, I streamed content using a Chromecast, either from a phone app of from the PC.
Speaking of which, my PC is vintage 2012 or so, Win7 native upgraded to Win10, and it still works. I’ve heard of even older machines running even older systems that are still around. I’m sure if I dug out my old Vista PC (c.2007), it would run (Hm. I wonder if it can be upgraded to Win10?)
I expect Microsoft and OG PC makers to correct this error soon.
Yes. There is.
STOP BUYING INTERNET-CONNECTED PILLOW CASES!
@James Joyner:
My point was systems are always going to go down, it doesn’t matter how many safeguards are in place. Up to a point you can build in some redundancy, but that comes at a cost, usually higher than the loss from a disruption.
My favorite internet to control things at home story happened while I was still working. I was at a conference in Orlando. A colleague who is an early adopter of whatever tech toy is out there was turning down the thermostat in his home three thousand miles away. His wife who was actually in the house and exposed to the local climate would turn it up. The time zone difference meant that she awakened in a cold house. He would check the temperature in the house between the presentations,and I suspect during boring portions of the meeting, which resulted in a seesaw battle between the on site wife and the far away controller husband. At dinner that evening they had a phone conversation. I was not snooping, but he was quite loud. The internet of things may be hazardous to your marriage. They should put a label on it.
Got a smart thermostat because our daughter loves tech; have since turned off all the ‘smart’ features because it is not smart. Got a car with the ability to keep me in a lane; turned it off because it, too, was actually quite stupid. Have a TV with all sorts of smart features I’ve never even bothered to learn about since none of them do anything useful. Have an Apple watch I stopped using after a couple of months because it was almost entirely useless and kept telling me (and family) that I had taken a hard fall, when I had not.
My toaster is not smart, but really doesn’t need to be because it’s a toaster.
Well, I’d have to say that most of y’all are severely engaged in hindsight bias.
AND, the reason that failure modes have not been investigated, and contingencies built for them is that that is hard and expensive.
It is not impossible. It’s not even unclear what to do. It is simply a lot of work, and a lot of pain to make sure it’s working right.
Once upon a time, I worked for a system vendor. I worked in software, we had a central repository for our source code. This, by the way, is insanely valuable to us as a company. A couple of times this went down, and nobody could really do any work for a few hours. This was accepted as the price of doing business.
The other thing that happened is that disk storage had a head crash, and the data on it was lost. No problem, we thought, we do nightly backups. And then we discovered that the nightly incremental backups weren’t working correctly, and we lost two weeks worth of checkins. That’s two weeks times 300 programmers who are top of the field and paid well. Also a schedule hit.
Because, it seems, nobody had checked to see if the incremental backups worked, by doing restores from them as a routine drill.
In part this comes from a sense that “it’s just copying files, it can’t be hard, and nothing can go wrong”. Insufficient paranoia, coupled with schedule and cost pressure.
Some organizations learn from this kind of thing. I know ours did. AWS might, but the people who really need to learn from this are people like Eight Sleep, who really need to have a design that gracefully degrades
(“Graceful Degradation” we used to say, should be the name of a band)
I’m just wondering how much disposable income one must have to pay an annual subscription for an internet-enabled pillowcase cover.
@Jay L. Gischer:
Back when I worked for a systems integrator, a most frequently asked question came from the business side regarding system uptime and redundancy. The question was asked so frequently that I simply provided tiered options for keeping everything running. No one bought the tier one option that would protect against a black swan event, to expensive, everyone bought redundant HDD as those were the most likely point of failure and also the cheapest to protect against.
PS–I recently read Culpability, a novel that follows five members of a family who were involved in a car crash while in an advanced AI-powered minivan. Interesting book, and the title is very appropriate.
@Jay L. Gischer:
We’ve had the backups fail. It was an external HD connected to the file server (which was also a working PC), that was supposed to backup the server every day. Sometimes it did not. We also tried copying files manually to CDs, then DVDs, then USB thumb drives. Now we use “the cloud.”
The latter has been good. But now and then it does weird, unexpected stuff. Monday I took half a day to work on a product listing. Tuesday morning it wasn’t there for some reason. I had to do it over. for now, I’, checking the time stamp on all working files to see they are backed up to the cloud. Come now, we use auto-save, and I’m on a laptop plugged to a power outlet. I should be proof against power outages as well…
I need to check whether we keep local copies of the cloud files. At home I do just that, because I’m not trusting some MS service with my fiction and other things. So it goes to one Drive, yes, but also to my hard drive (and I back up to portable media when I remember to do so; I should make it a weekly ritual).
All that aside, if you make a product that depends on some third party service to work right, in the case of the internet mattress covers two third parties (AWS and the local ISP for each user), it should come with a failed connection mode of some sort. Audiobook apps do this. If they have no internet connection, they tell you so and instruct you to use the books you downloaded, if any.
@Jay L. Gischer: Condolences.
Long ago, I worked in a lab where we were buying a new minicomputer (I said long ago, did I not?) The corporate computer center, which cared for our first minicomputer, was highly upset that we put the new machine in our lab space and did our own backups. Once the pissing contest had been elevated to an appropriate level, I went to the meeting. “We’ve needed data restored from backup twice,” I told them. “The first time we were told the backup hadn’t been done for three weeks because the backup media was broken. The second time we were told the backup hadn’t been done for four weeks because the person who did that left the company and the job wasn’t reassigned yet. Give us a call when you can demonstrate that you haven’t missed a nightly backup for, say, a year.”
As I recall, the computer center manager lost their job over that issue.
@Jen:
I think it’s a mattress cover. It gets advertised a lot on Youtube. I ignore commercials on principle, and skip them when the option appears. I’ve also seen the odd review about them here and there. The main benefit seems to be they can keep the mattress cool or warm, according to what the user directs.
I didn’t even think about it, figuring it would consume a lot of electricity and require a water hookup of some sort, and might be noisy to boot. Then I saw the price. IMO, it would be better to get AC in the bedroom.
@Kathy:
This excerpt is directly from the snippet above:
I still think it’s silly.
@Kathy: I’m still running ancient software, specifically, Microsoft Money Sunset Edition (2010). Really don’t want to track my money in an online database so I keep this alive. Every major Microsoft Windows upgrade (like from 10-11) I have to go in and tweak it to run. But it is still kicking.
@Jen:
Very silly.
I appreciate a cool pillow on a warm night, but it lasts like, what, a minute? I cant imagine paying for a gel pillow (purported to keep cooler longer), much less thousands and a subscription for a “smart” pillow cover.
I wonder how different things look like when one has effectively unlimited amounts of money to spend…
@Scott:
I had a 90s era shareware backgammon game that followed me from Windows 98 to Win10. It ran every time without issues. I lost it in 2016 when my current PC crashed and I had to re-format the hard drive.
It might still be in the old Vista PC. I really should dig it out.
The common sense approach in tech is for companies to look at this outage, realize how vulnerable they are to a single point of failure and try to build in enough redundancy that they can continue to function even if us-east-1 goes down. (That’s the most common AWS datacenter, and it’s where the problem was)
I think that’s the wrong lesson. Odds are your services don’t matter that much, you should shove it all into us-east-1 and rest assured that if that datacenter goes down your customers will be more upset that some other service they depend on more is down, and that amazon will be scrambling to fix things. Just save yourself a lot of time, money and hassle.
No one ever thinks I have good ideas though.
Not that long ago, hardware tended to wear out before it was made obsolescent by technology. I know “inbuilt obsolescence” was a thing back in the day but that was aimed at manufacturers who released products knowing full well that better features would be in the next model. The earlier products still worked. Disc brakes and fuel injection didn’t stop our older cars’ drums and carburettors working as intended.
That’s changing. Last year Australians had to throw out millions of perfectly good cell phones because the carriers (with government appoval) switched off the 3G network. Older phones (as in less than 5 years in some cases) could no longer call or text. I dumped my two “premium” phones in disgust and bought the cheapest replacement I could find. Now tens (hundreds?) of millions of people around the world have been informed their computers are about to become increasingly vulnerable to hacking, because Microsoft is no longer going to update Windows 10.
Quite apart from the inconvenience and cost to ordinary users, the unnecessary waste generated by “progress”* like this must be staggering.
*I’m sure Windows 11 offers superior performance to Windows 10 for some users, but I have desktops running each one, and for my purposes they are identical. The only annoyance in upgrading the desktop that was compatible with Windows 11 was that I had to relearn where many functions and settings were because they had been pointlessly changed.
@Ken_L:
Microsoft traditionally stops supporting older, discontinued operating systems. The scurvy trick they threw this time is the hardware requirements that many older systems don’t meet, and that are not strictly necessary for Win11 to run.
It’s not as though I expect that a PC running Win98 should be able to run Win11. But one running Win7 that upgraded to 10, and also one that ran Win8.1 and upgraded to 10, should be able to run the newest version.
BTW, it may not matter now, but some PCs can extend update support for one year for free. This video shows how
My nephew (er, his real estate wife) got an IOT Fridge where with an app they could see inside the fridge. Why? Another thing to break.
@Michael Reynolds: @Kathy: For the most part, I prefer ChromeCast to the built-in smart apps. But the former sometimes has lag during live events and can lose the connection, so that one can’t pause. And there are apps on the TV that have cool features. For example, YouTubeTV lets me watch up to four games at the same time. That’s not quite as good as RedZone but it’s cool if you’re just killing time waiting for your team’s game to start.
@Ken_L: It’s not about the increased capabilities or performance of Windows 11, it’s about several things:
1) Employing legacy programmers who are skilled and smart and don’t want to work on old technology their entire careers.
2) Architectural decisions made during the early days of old technologies that constrain the response to security problems, or desired features in programs running on the OS.
3) Increasing complexity as time goes on. Multiple versions of multiple products – some of which didn’t even exist when Windows 10 came out – are now expected to run, and co-habitate, on the older version. The support matrix expands geometrically with every new version of every new product installed on top of Windows.
In short, new versions of Windows are a code-fork, and supporting old stuff becomes very expensive over time.
And that expense does not drive any new revenue for the company.
@James Joyner:
I got a Chromecast long ago, because it was the only way I had then to stream video to the TV (though I suppose I might have run an HDMI cable from the PC to the TV; I vaguely recall thinking about it).
Post late 2021 when I got a “smart” TV, I run the onboard apps. however, sometimes the TV apps suck, and I resort to the Chromecast and PC website or phone app. For instance, the early Paramount+ app on the TV didn’t let me choose audio language for streaming. It only played dialogue dubbed to Spanish (did I ever mention I absolutely hate dubbing?) So I streamed it from the PC.