The Long Now of the Web: Inside the Internet Archive’s Fight Against Forgetting – HackerNoon

New Story, 1,290 reads

The Long Now of the Web: Inside the Internet Archive’s Fight Against Forgetting

by Bruce Li, January 12th, 2026

A Comprehensive Engineering and Operational Analysis of the Internet Archive

Introduction: The Hum of History in the Fog

If you stand quietly in the nave of the former Christian Science church on Funston Avenue in San Francisco’s Richmond District, you can hear the sound of the internet breathing. It is not the chaotic screech of a dial-up modem or the ping of a notification, but a steady, industrial hum—a low-frequency thrum generated by hundreds of spinning hard drives and the high-velocity fans that cool them. This is the headquarters of the Internet Archive, a non-profit library that has taken on the Sisyphean task of recording the entire digital history of human civilization.

Internet Archive’s office in San Francisco

Here, amidst the repurposed neoclassical columns and wooden pews of a building constructed to worship a different kind of permanence, lies the physical manifestation of the “virtual” world. We tend to think of the internet as an ethereal cloud, a place without geography or mass. But in this building, the internet has weight. It has heat. It requires electricity, maintenance, and a constant battle against the second law of thermodynamics. As of late 2025, this machine—collectively known as the Wayback Machine—has archived over one trillion web pages.1 It holds 99 petabytes of unique data, a number that expands to over 212 petabytes when accounting for backups and redundancy.3

The scale of the operation is staggering, but the engineering challenge is even deeper. How do you build a machine that can ingest the sprawling, dynamic, and ever-changing World Wide Web in real-time? How do you store that data for centuries when the average hard drive lasts only a few years? And perhaps most critically, how do you pay for the electricity, the bandwidth, and the legal defense funds required to keep the lights on in an era where copyright law and digital preservation are locked in a high-stakes collision?

This report delves into the mechanics of the Internet Archive with the precision of a teardown. We will strip back the chassis to examine the custom-built PetaBox servers that heat the building without air conditioning. We will trace the evolution of the web crawlers—from the early tape-based dumps of Alexa Internet to the sophisticated browser-based bots of 2025. We will analyze the financial ledger of this non-profit giant, exploring how it survives on a budget that is a rounding error for its Silicon Valley neighbors. And finally, we will look to the future, where the “Decentralized Web” (DWeb) promises to fragment the Archive into a million pieces to ensure it can never be destroyed.5

To understand the Archive is to understand the physical reality of digital memory. It is a story of 20,000 hard drives, 45 miles of cabling, and a vision that began in 1996 with a simple, audacious goal: “Universal Access to All Knowledge”.7

Part I: The Thermodynamics of Memory

The PetaBox Architecture: Engineering for Density and Heat

The heart of the Internet Archive is the PetaBox, a storage server custom-designed by the Archive’s staff to solve a specific problem: storing massive amounts of data with minimal power consumption and heat generation. In the early 2000s, off-the-shelf enterprise storage solutions from giants like EMC or NetApp were prohibitively expensive and power-hungry. They were designed for high-speed transactional data—like banking systems or stock exchanges—where milliseconds of latency matter. Archival storage, however, has different requirements. It needs to be dense, cheap, and low-power.

Brewster Kahle, founder of Internet Archive (with the PetaBox behind him)

Brewster Kahle, the Archive’s founder and a computer engineer who had previously founded the supercomputer company Thinking Machines, approached the problem with a different philosophy. Instead of high-performance RAID arrays, the Archive built the PetaBox using consumer-grade parts. The design philosophy was radical for its time: use “Just a Bunch of Disks” (JBOD) rather than expensive RAID controllers, and handle data redundancy via software rather than hardware.

Editor’s Note: Read the rest of the story, at the below link.

Continue/Read Original Article Here: The Long Now of the Web: Inside the Internet Archive’s Fight Against Forgetting | HackerNoon

Inside the old church where one trillion webpages are stored – CNN Business

Inside the old church where one trillion webpages are being saved

By Hadas Gold, Updated 23 hr ago

See inside the old San Francisco church that houses nearly all of the internet’s history…

San Francisco  —  Just blocks from the Presidio of San Francisco, the national park at the base of the Golden Gate Bridge, stands a gleaming white building, its façade adorned with eight striking gothic columns.

But what was once the home of a Christian Scientist church, is now the holy grail of Internet history — the Internet Archive, a non-profit library run by a group of software engineers and librarians, who for nearly 30 years have been saving the web one page at a time.

Inside the stained-glass-adorned sanctuary, the sounds of church sermons have been replaced by the hum of servers, where the Internet Archive’s Wayback Machine preserves web pages.

The Wayback Machine, a tool used by millions every day, has proven critical for academics and journalists searching for historical information on what corporations, people and governments have published online in the past, long after their websites have been updated or changed.

For many, the Wayback Machine is like a living history of the internet, and it just logged its trillionth page last month.

Archiving the web is more important and more challenging than ever before. The White House in January ordered vast amounts of government webpages to be taken down. Meanwhile, artificial intelligence is blurring the line between what’s real and what’s artificially generated — in some ways replacing the need to visit websites entirely. And more of the internet is now hidden behind paywalls or tucked in conversations with AI chatbots.

It’s the Internet Archive’s job to figure out how to preserve it all.

“We are here to try to provide a record of what happened, so that people can learn and build on that to build a better future, or to build new ideas that are worthy of being in the (Internet Archive’s) library,” said Internet Archive founder Brewster Kahle.

The internet’s library

Kahle created the archive in 1996 when a year’s worth of saved pages could fit on about 2 terabytes worth of hard drives, the amount of storage you can get today in an iPhone. Now, the archive is saving closer to 150 terabytes, or hundreds of millions worth of web pages, per day.

Kahle is the driving force and personality behind the archive, with the exuberance and energy of your favorite science teacher and like an evangelist whose religion is libraries and technology. Sitting for an interview on the original wooden pews of the church, Kahle said he was inspired to purchase the building because it resembles the group’s logo. But more importantly, he said it’s a symbol of permanence and a reference to the Library of Alexandria in Egypt.

“That was the first time somebody tried to go and collect everything ever written by humans,” Kahle said. “Of course, now that place is the internet, and the Internet Archive serves the whole internet as a library.”

The Wayback Machine tool does more than just screenshot the page. It also saves the technical architecture — the HTML, CSS, JavaScript codes and more — so that it can attempt to “replay the page as it existed” even if the server is no longer functioning, said Wayback Machine Director Mark Graham.

The rise of artificial intelligence and AI chatbots means the Internet Archive is changing how it records the history of the internet. In addition to web pages, the Internet Archive now captures AI-generated content, like ChatGPT answers and those summaries that appear at the top of Google search results.

 Referred by: Library Link of the Day
http://www.tk421.net/librarylink/  (archive, rss, subscribe options)

Continue/Read Original Article Here: Inside the old church where one trillion webpages are stored | CNN Business

'The Truth Is Paywalled.' Internet Vets Lament the State of the 'Open' Web

"The North Star in this system is that everything should be able to connect to everything else. That openness is really vital to the utility of the internet." - Vint Cerf

#internet #VintCerf #BrewsterKahle #CindyCohn #JonStokes #technology #tech

https://www.pcmag.com/news/the-truth-is-paywalled-internet-vets-lament-the-state-of-the-open-web

Weekly output: phone plans, Nvidia keynote, passkey adoption, Bending Spoons buys AOL, SpaceX simplifying Starship lander, Internet luminaries on the open Web

This is not going to be a great week for normal sleep cycles: Tuesday, I will wake up at around 4 a.m. to spend a 15-plus hour shift working as an election officer for Arlington, and then Wednesday I’m off to Dulles Airport for this year’s final business trip across the Atlantic. I’m departing for Web Summit in Lisbon several days early because the organizers of another conference, the Mozilla Festival, offered a press pass and a travel stipend to cover that event in Barcelona. I’ve heard good things about this conference over the years, so accepting an invitation to spend a few days in one of my favorite cities in Europe was an easy call.

In addition to what you see below, Patreon readers got a detailed recap of how this past week’s event-packed schedule left its own series of dents in my calendar.

10/28/2025: The Best Cell Phone Plans, Wirecutter

This was going to be a modest update to the guide that I’ve been maintaining since 2014, but T-Mobile jacking up prices while AT&T and Verizon inflicted more modest rate hikes led to us dethroning T-Mo on cost grounds and handing our “for most people” pick to AT&T, which has advanced its own 5G network considerably.

10/28/2025: In DC, Nvidia CEO Touts New AI Partnerships, Goes a Little MAGA, PCMag

Heading into Nvidia’s conference, I was worried that CEO Jensen Huang would go into the weeds about the finer points of GPU architecture. Instead, he used this nearly two-hour keynote to jump from topic to topic without getting into too much detail about any of them–and kept coming back to opportunities to praise President Trump.

10/29/2025: Passkey Adoption Sees Striking Progress, With One Obvious Leader, PCMag

I struggled to get this written at the end of a long workday, resulting in my getting some nuances wrong that required updating the post the next morning.

11/1/2025: Serial Dot-Com Purchaser Bending Spoons to Buy AOL, But Why?, PCMag

Writing about AOL in 2025 makes me feel so old, but as one of PCMag’s graybeards I had to cover the news of Bending Spoons buying the company that once ruled the online world. I got to this story a day after it broke, so I turned that lag into an opportunity to expand the piece with some quotes from a publicist for that Italian firm and from a podcast interview of its CEO Luca Ferrari last year

11/1/2025: After Elon Tantrum, SpaceX Now Prepping ‘Simplified’ Starship-Based Lunar Lander, PCMag

Since I wrote about Elon Musk’s childish reaction to NASA’s understandable concern over the pace of its Human Landing System work, I had to reach for a keyboard to cover SpaceX’s grown-up corporate response.

11/1/2025: ‘The Truth Is Paywalled.’ Internet Vets Lament the State of the ‘Open’ Web, PCMag

This Monday-evening panel was one of the first items on my calendar this week, but having event after event after event follow it led to me not writing it up until Thursday night. Once again, it was a serious treat to hear some of the Internet’s founding figures talk about the state of the thing they invented.

#AmericaOnline #AOL #ArtemisIII #ATT #BendingSpoons #BrewsterKahle #CindyCohn #Dashlane #FoundationForAmericanInnovation #HumanLandingSystem #JensenHuang #Nvidia #NvidiaGTCDC #passkeyExport #passkeys #phonePlans #smartphonePlans #SpaceX #TMobile #unlimitedData #verizon #VintCerf

https://archive.org/donate?origin=iawww-TopNavDonateButton
A Message from #InternetArchive Founder, #BrewsterKahle:

„Dear Patron,
For more than two and a half decades, we have collected, preserved, and shared our digital #cultural #artifacts. Thanks to the generosity of our patrons, the Internet Archive has grown from a small #preservation project into a vast library that serves millions of people each year. Our work has impacted the lives of so many of our users who value free and open access to #information. (1/4)

#Stross's "Slow AI" as a theory of the corporation puts in an appearance at #BradDelong's blog today.. (A neater formulation of #BrewsterKahle's "1st generation AI").

https://braddelong.substack.com/p/hoisted-from-the-archives-a-return

"Six weeks into the new administration, Wayback Machine director Graham said, the Internet Archive had cataloged some 73,000 web pages that had existed on U.S. government websites that were expunged after Trump's inauguration."

Support these people!

www.npr.org/2025/03/23/nx-s1-5326573/internet-archive-wayback

#InternetArchive #BrewsterKahle #WaybackMachine #MarkGraham #Archives #Libraries #DigitalHistory #LinkRot