A Ritual to Secure the Internet

The Kaminsky Bug

The Master Key

Crypto Officers

The Ritual

Mistakes are made

Backup Key Holders

The Ceremony comes to a close

Strangers with Keys

Daina Bouquin 0:20

Every night for over seven hundred years, the exact same thing has happened at the Tower of London. In the dark, the chief yeoman warder walks to the outer gates. Challenges are called out into the night air. Passwords are exchanged. The heavy wooden doors are locked, and the keys are secured until morning. It has happened through plagues. It has happened while bombs fell during the Blitz. The ritual has never been skipped. We do this because humans have always understood something about security. Certain things require a ritual, not just because the ritual makes us safe, but because it makes trust visible. Back in 2008, a security researcher named Dan Kaminsky realized that the internet's core directory was unprotected. It was a well without a lid, vulnerable to anyone who wanted to secretly poison the water and redirect the world's digital traffic. He forced the architects of the web to build a heavy cryptographic lid to protect the directory from poisoning. A master key to lock the internet's route. That story is its own episode and it's worth your time. But a master key creates a new terrifying problem. If you give it to one person, they become a target. If you give it to a single government, it becomes a weapon of politics. So they broke the key apart and scattered the fragments. I'm Daina Bouquin, and this is Lore in the Machine. There are only 14 people in the world who hold the title crypto officer. They are not spies or government agents. They are trusted volunteers, like Swedish internet expert Anne-Marie Eklund Löwinder, who stored a piece of the key on a long metal chain inside a wooden puzzle box with a hidden lock. Her son made the box in his workshop. He also made furniture. Four times a year, a handful of crypto officers pack their bags, get on airplanes, and travel to a secure facility in either Culpeper, Virginia, or El Segundo, California. They go there to perform a ritual. If you were to dream up a ritual to protect the global internet, you might imagine a sci-fi fortress, an unassailable tower with an arsenal. But the reality is much stranger. It is a bureaucratic fever dream pieced together from mathematics, paranoia, and plastic. Imagine walking into the El Segundo facility. The walls are beige and unremarkable. You hand over your government ID. Your bag is searched and you are given a badge. Then you enter a man trap. It's a small room with doors at both ends. Only one door can open at a time. To get through, you need a pin, a card, and your hand. To exit, you need them again. Now you wait in a sterile space where lunch is being served. It looks like a doctor's waiting room. But it has an Atari arcade machine. The people who designed this ceremony actually wrote a 5% "dishonesty rate" into its mathematical specifications. There are over 100 highly scripted actions, and the entire system is built on the assumption that someone in this room is secretly a traitor. When it's time, the ceremony begins. There are no passwords exchanged at the door, no challenges called out into the air. It's more like a heist movie run in reverse. To enter the hallway, a staff member swipes an access card and presses their palm against a scanner. To enter the main room, they lean into the red glow of a retina scanner. When the heavy door clicks shut behind you, you are standing inside a Faraday cage, an enclosure that shields its interior from external electric fields and electromagnetic radiation. The entire room is completely signal proof. Nothing can enter or leave. Inside this room, there is a large metal cage containing two heavy safes. Once the safe controllers are brought in, the steel door of the first safe is pulled open. Inside are safe deposit boxes, each requiring two physical metal keys to be turned at the exact same time. Inside the box is a smart card. It rests in a hard plastic case, sealed inside a tamper evident bag. The plastic case was added to the ceremony. After someone realized that a phantom thief could theoretically slip a microscopic needle through the plastic bag to manipulate the card without leaving a tear. The second safe is opened next. Inside is the master lockbox. It is designed to self-destruct. If someone tries to cut it, freeze it, heat it, or even shake it too hard, it will instantly wipe its own memory. Next to it is a laptop. This laptop has no hard drive. It has no battery. It has no memory of its own. It's completely air gapped, ensuring the master key can never ever touch the internet. It doesn't even have a tiny backup battery to keep its internal clock running. When it's plugged into the wall, the time has to be set manually using an isolated drifting wall clock that has been hanging in the room for over a decade. The ceremony script actually refers to this clock as the quote, "reasonably accurate clock visible to all in Tier 4 (Key Ceremony Room)". These are hyper serious security measures that border on hallucination. But because they're performed by humans, they're imperfect. During a ceremony in 2014, a security controller accidentally slammed the door of the safe too hard. It triggered a seismic sensor, which immediately triggered the automatic door locks. The administrators and the key holders were suddenly trapped inside the metal cage. They stood there in quiet panic for six minutes until someone finally triggered an evacuation alarm. The sirens blared and everyone piled into the hallway. They ate Oreos and Cheez-Its until the system reset. In 2020, the sophisticated lock on one of the safes simply broke. A certified locksmith had to be called in, and the people holding the keys to the internet had to sit around for over 20 hours while a guy with a drill tried to bust open the door. There are also backup key holders. People entrusted with smart cards containing a fragment of code needed to rebuild the key generating machine from scratch in case something calamitous happens. Once a year, these people take a photograph of themselves holding their key next to that day's newspaper, just to confirm that all is well. The room also needs to be kept clean, but cleaners aren't allowed inside. Anne-Marie Eklund Löwinder was known for meticulously vacuuming the room with a $20 dustbuster. After the room is clean and the cage is unlocked, everything is laid out on a table. Cameras are recording. The encrypted requests are loaded via a USB drive. The smart cards are inserted. This process takes time. In all, the ceremony usually takes roughly four to five hours. But then it's time for the final command. The Ceremony Administrator types a single letter. Y for yes. With that single keystroke, the dramatic portion of the ceremony ends. The root of the internet is cryptographically signed. The directory is authenticated. And the digital world is secured for a few more months. The logs are printed. The smart cards are sealed back into new tamper-evident bags. The heavy safes clang shut and are locked. And the crypto officers scatter back across the globe, returning to their normal lives. We like to think the internet is a machine. An indestructible web built on software, mathematics, and fiber optic cables. But it isn't. It is a fragile thing held together by strangers. Strangers who pack their bags four times a year, who fly across oceans to sit in a windowless room, who submit their eyes to scanners, and turn metal keys in unison. They don't do it because the code requires an audience. They do it because we do. Because we need to know that someone is out there in the dark, calling out the passwords and making sure the gates are locked until morning. I'm Daina Bouquin, and this is Lore in the Machine.