Reflections as the Internet Archive turns 25

307 points12
tkgally11 hours ago

In his reflections, Brewster Kahle mentions his goal of creating “a library available to anybody, anywhere in the world.” He doesn’t mention, though, the costs of making that library available to the world for free or the fact that the Internet Archive accepts donations. So I will:

raybb8 hours ago

Donations are fantastic but if you have engineering (or project management, design, etc) skills spending just 1 hour a week contributing to their open source goes a very long way!

Open Library in particular has a very active repo with lots of volunteers, a weekly community call, and a rather accessible codebase.

If anyone knows webpack well would LOVE to have this dev-facing issue resolve to auto reload CSS

dailyanchovy6 hours ago

Alright, I'm new to pull requests but I had a go at it!

state_less2 hours ago

I sent my bones or clams or whatever you call them.

Will send moore when I have more and when I've learned to be more generous. It's good to know that you're near Internet Archive.

But, oh, what a wonderful feeling

Just to know that you are near

Sets my a heart a-reeling

From my toes up to my ears

-Bob Dylan, The man in me

mkaufman28 minutes ago

state_less sounds like a Little Lebowski Urban Achiever.

walterbell9 hours ago

Anyone know the relative budgets/donations/staff of Wikipedia vs.

tkgally8 hours ago
jdc33 minutes ago

Wow, never thought I'd see Wikimedia at more than 4x the budget of Mozilla!

joe_the_user11 hours ago

Nice they're there. At the same time, it's amazingly easy for content to be removed from there - if someone objects or even if things are murky.

For example, all content from the old ezboard site was been removed based on the configuration of the current URL owners' robots.txt, and current URL owner is just a domain parker. Ezboard hosted a lot of content back in the day.

1vuio0pswjnm79 hours ago

This is an old problem I could have sworn there were promises they were going to change their procedures.

The question I have is how fast is the content removed after the domain name registration changes, i.e., is there is a window of time between the appearance of a new robots.txt and the next scheduled crawl, and if so, is it be possible to "rescue" the content, as ArchiveTeam would do, during that window, before it disappears.

If this is possible, there could be a service for monitoring changes to domain name registrations for sites that have large amounts of historical content. I would happily volunteer to set up such a service.

joe_the_user8 hours ago

Well, "complain on hn" has been a way to get stuff from Google. Maybe someone at archive will notice this thread.

SilverRed11 hours ago

Hopefully it is just hidden and not deleted. But this is the main reason why alternative archive sites exists which ignore the original posters requests. Frequently used to archive posts from public figures which are suspected to be attempted to be scrubbed later.

techrat7 hours ago

> Hopefully it is just hidden and not deleted.

Hidden. Even when you request for them to remove stuff.

Had domain, stuff got archived, asked for them to remove it, added robots.txt. Domain lapsed. Someone else picked it up. their robots.txt now permissive, old stuff that I requested for them to remove is now visible.

throwslackforce3 hours ago

That's insidious. Does it even make sense to revive data that has been removed based on current configuration?

Even if the owner is the same, allowing the site to be archived going forward isn't the same thing as permitting it retroactively.

mavhc4 hours ago

Wonder how you'd overcome that flaw, is there a history of domain name ownership?

newswasboring3 hours ago
account423 hours ago

Maybe we need a

fwn8 hours ago

As far as I was able to experience it's just hidden and not deleted.

I have to keep an old domain indefinitely to host a robots.txt just to keep sensitive personal data hidden that little me foolishly published on the open internet.

But I'm not complaining. The internet archive is a great gift. Using it with a bookmarklet really feels like a super power.

mercora6 hours ago

it sounds a lot like this would need some kind of delegation mechanism where you could point to a different URL in-time before abandoning the place. or maybe some kind of sealing using a cryptographic function that lets you proof your are the owner of the current/previous content and also would proof you are not the owner of the newer content while this proof could be used to release the ban if ever needed.

joe_the_user11 hours ago

Got any examples of these alternative archives?

SilverRed10 hours ago

This is the main one

From the FAQ, they do not respect robots.txt since they only archive on request by a user and they do not remove archives unless they contain illegal content.

1vuio0pswjnm78 hours ago

HN commenters like to use but I always wonder if people are aware that is (a) blocked in some countries^1 and (b) may block access to itself in a country if it feels threatened.^2

There is also the issue of EDNS subnet.^3 tries to require it; it wants to know what location a request is coming from. In addition to EDNS, inserts the IP address and geolocation of the incoming request into the HTML of the returned page as a tracking pixel.^4

Thus does some things does not do besides just ignoring robots.txt

One of the things does that does not do is that inserts an HTTP response header intended to disable Chrome FLoC.^5 I add this header for all sites in a local proxy; however I do not see many sites adding it as a courtesy. Thanks for doing that.





5. permissions-policy: interest-cohort=()

mellosouls7 hours ago
v0x10 hours ago

There’s but I get the sense that the major use case for that is getting around paywalls as opposed to permanently archiving a page - indeed, since they host content that the site owner probably doesn’t want them to, it would stand to reason the service would not be likely to stand the yet of time. But I could be wrong.

Jiro8 hours ago

They actually posted about this in 2017: . At the time it sounded like they might change their robots.txt policy. I guess they never followed up on it.

(I checked and ezboard is still excluded.)

elric4 hours ago

Without trying to be contrarian, I don't think that everything should be archived. Random tweets, random blog posts, random personal web sites. Let them wither and die and be forgotten. Notable content by notable people? Sure.

Everyone else ought to have the right to be forgotten, including some drunk tweet they wrote 10 years ago and regret, or an old personal page which contained too much PII.

Archive no longer has a way to opt-out, which is bad enough, but I still think they should be opt-in.

X6S1x6Okd1st29 minutes ago

Historians spend a lot of time pouring over minutia from non-notable people. There is plenty about the world that doesn't seem worth writing down, but can be intuited from tangential texts.

The costs seems low enough to just keep it.

jl63 hours ago

Perhaps the Internet Archive could do more to help people who find their personal/sensitive/embarrassing content made available in perpetuity (I’m not sure exactly what they could do), but it’s incredibly valuable to have archiving on by default. The voice of un-notable people is underrepresented in every field of study, and the voice of notable people tends to get preserved in other ways anyway.

jjkaczor1 hour ago

It is helpful to get the perspectives of "un-notable" people from a historical perspective.

For example - the graffiti at Pompeii is interesting (and is pretty much at the same "quality bar" as Twitter):

jacquesm4 hours ago

You never know in advance what will and what will not be worthwhile archiving, you only know that at some unspecified point in the future.

elric2 hours ago

That's lovely from the perspective of a historian 200 years in the future. But it does not nothing to alleviate the pain of people living in the present. People whose prospective employers comb over their embarrassing past. Or bullies. Or any other number of evildoers whose life is made easier by unfettered access to indelible information.

generationP3 hours ago

Who decides what "notable" is? I frequently use the Archive to find old academic grey lit (preprints, lecture notes, newsgroup posts, etc.). Much of it is on "random" blog posts and personal websites. Even the authors aren't usually notable by Wikipedia standards. Yeah, there is some PII on those pages, but also treasures of useful information.

stared3 hours ago

Most of history we know is from the perspective of the wealthies 0.1%. Even though the Internet is still biased towards the wealthier and more educated, having a history of the wealthies 10% would be enormous progress.

DonHopkins3 hours ago

A search of youtube for "wayback machine" produces pages of stuff about the Internet Archive, and only the 24th result has anything to do with the origin of the term.

People who didn't spend their Saturday mornings glued in front of the TV screen as a child of the 1970's might not remember how American kids learned about history back then:

Peabody's Improbable History - Surrender of Cornwallis

Peabody and Sherman travel back to October 19, 1781 to witness when Cornwallis surrendered for Washington. However, when they got there, then he didn't show up.

spiritplumber7 hours ago

The Internet Archive location is beautiful. it's a church that has been partially turned into a server farm. Big Neuromancer energy when you go inside and look.

DoingIsLearning6 hours ago
TekMol3 hours ago

How can IA just go out there and copy+republish the content of others and get away with it?

Aren't they breaching copyright on a massive massive scale?

ghaff2 hours ago

Because the vast majority of people don't care if someone archives something they've made public. And the Internet Archive bends over (to some too far) backwards with robots.txt to exclude anything someone wants removed from public view.

dleslie11 hours ago

I love the Internet Archive; I worry that its utility will wane as content becomes more dynamic than static. What does it mean to archive the experience of scrolling through a social feed?

petertodd9 hours ago

The paid, legal-oriented, archiving service that Harvard Law runs actually lets you upload your own PDFs and screenshots in addition to allowing's bots capture webpages. Of course, since you could upload anything the difference is made clear in the UI.

In a legal context, simply attesting to the validity of a screenshot is really common. So when that functionality is used is operating more as a permanent file storage service than a trusted archive.

Regardless, this does go a long way to solving the problem of dynamic sites.

ZeroGravitas6 hours ago

This is a bit of a tangent, but there was mention of non-advertising based funding models.

Is anyone working on an advertising model that achieves the basic goal of advertising, but without the centralising aspect which seems to be the root cause of many of the issues? Giant monopolies are always going to subvert regulation but the same industry as disconnected units might be easier to police. Obviously you can just try to split them up or limit their size with regulation after the fact but a good technical basis might help out.

Does web advertising just not make sense unless you can amass lots of private user data and track people across the web? If so can we subcontract that data to smaller companies we can trust with our data and effectively punish if they break the rules?

causi2 hours ago

As a question related to archival, what's the best tool for local archiving? HTTRACK is getting long in the tooth and it just work for all the dynamic content on modern web pages.

akkartik4 hours ago

I wish IA hosted Usenet archives. Even if they stopped at year 2000 and didn't update further.

generationP3 hours ago

They host some:

As far as my own old posts are concerned, it looks complete :) But it isn't easily findable or searchable; the intended way of interaction is apparently to download an entire hierarchy and grep.

ezequiel-garzon10 hours ago

This library would be available not only to those who could pay the $1 per minute that LexusNexus charged

Any idea on what LexusNexus is, or was back then? Thanks!

fumeux_fume10 hours ago

I think it's a reference to LexisNexis which still exists today in a similar capacity. Back in the day (1990s), it was an online service to search for and read virtually any legal or news article among many other things. It was very expense, but was usually access thru a corporate or educational institution's license.

Mountain_Skies7 hours ago

If you live in the US, they have a file on you and it's probably very thick. A credit report will run a couple of pages long, the LexusNexus report on me (a complete nobody) cost them about $8 to mail. It contained many errors, mostly about property I don't own but also about an insurance claim I never made.

Like your credit report, you can get a free copy by writing them and requesting a copy. IIRC when I did it a few years ago, I had to make the request in writing, I wasn't able to order it online, at least not for free.

jacquesm4 hours ago

They ought to be liable for those errors.

lifekaizen10 hours ago

Very expensive research tool places like large law firms would use.

endisneigh11 hours ago

I'm curious - how does get around DMCA? I assume it's because Fair Use and the fact that they're a not-for-profit, but more details would be great.

Other sites like (which I guess is a for-profit) entity don't really allow you to get around paywalls the way the Wayback Machine does.

As someone interested in building a site that gets around paywalls for semi-educational purposes I'm curious if anyone has details!

pabs310 hours ago

IA are a designated library, which confers a set of privileges under US copyright law.

DoingIsLearning9 hours ago

Yes as a non-US based user this is one of the most amazing things for me with IA.

in my head IA was just the wayback machine with cached pages. But during the pandemic I realized that there is a plethora of actual books that one can checkout with far less friction than in a standard library.

It is such a cool concept, also there are IA satellite projects (not sure if they are owned by or in partnership), for different non-english languages e.g. so you can have the same plethora of content in other non-english languages as well.

ghaff2 hours ago

A very limited set of privileges. And AFAIK it's only a "designated library" in California. I'm not sure there is such a thing at the federal level other than the Library of Congress.

Basically, most people are fine with what the Wayback Machine does and they'll take down any mirror that the domain owner asks them to.

toomuchtodo11 hours ago

Takedowns cause the content to be “darked.” It’s no longer publicly available, but still archived on disk until a future date.

joe_the_user11 hours ago

My impression follows whatever directions are in robots.txt. If those directions allow a site to be archived, they do it, otherwise they don't. That's from looking at not-archived sites, ezboard, which mention in another post.

Santosh8310 hours ago

And what about sites that do not have robots.txt? Does IA snapshot those?

nlitened9 hours ago

Yes. By default, if you put information on the web, it is assumed that you are okay with people seeing it.

shrubby8 hours ago

Pictures and touchscreens have ruined the internet ;-)

soheil11 hours ago

For anyone interested I made a quick and dirty way to pull up the archive of any URL by prefixing it with


fouc11 hours ago

Nice, thanks! Ever since I started using last year, I've always wanted something like this! Could be paired with a bookmarklet too.


alexislours7 hours ago

You have always been able to do this with, I’ve been using this bookmarklet for a quite some time.

javascript: (() => {"" + window.location.href, '_blank')})();

fouc5 hours ago

I meant I wanted to go directly to the most recent result in the wayback engine for the given page.

codethief5 hours ago is not related to the Internet Archive's Wayback Machine, though, is it?

fouc4 hours ago

It's not related, but it will give you a link there if it doesn't have the page archived.

johtso7 hours ago

Is there an advantage compared to just putting `` before the url?

fouc4 hours ago

Oh interesting, also works, didn't know that.

soheil1 hour ago

Didn't know that, thanks. I guess just shorter.

ipsum29 hours ago

a little confusing since arxiv refers to a popular research paper archive. Useful project though!

IfOnlyYouKnew9 hours ago

Big fan of the archive, especially openlibrary. But the lofty talk of “democratizing” this-or-that is a bit overblown: in authoritarian countries, the archive does zero democratizing, because it’s either just not available, or all the relevant individual documents are blocked.

For democratic countries that have democratically elected to restrict the distribution of some materials the society considers harmful, such as Germany with Nazi propaganda, the archive happily decides to undermine those clear, longstanding, and — disproving the single argument for free speech absolutism — not slippery-sloping anywhere over decades. Why? Because laws you disagree with are, apparently, illegitimate.

It probably helps their bold defense of all that is holy to intimately know that these really are democratic countries, which aren’t going to just send a wet team to dismember them, Saudis-style, or to spend millions on an elaborate plan whose only purpose is to let you live for another month, with a clear mind that has complete certainty that you will die, and who did it.

The anti-semitism and racism on, plus some copyright violations isn’t a byproduct of their “freedom”. It’s all of it. There are plenty of free hosts for video or documents, and an hour at minimum wage would pay for hosting quite a lot for quite a while for most of the archive’s audience. But the killer feature is immunity, through anonymity and DMCA’s Safe Harbour.

Sure, everyone here is only defending “free speech” and would never agree with the swastika-fetishists. Only, somehow, they never complain about ISIS having a hard time on Twitter, or porn being censored on Facebook. It’s the scans of Der Stürmer, especially of the 38 to 44 vintage, are the chosen symbols of “democratization”.


textfiles8 hours ago

"Big Fan" is doing a lot of heavy lifting there.

ZeroGravitas6 hours ago

I was intrigued by this comment, but couldn't really figure out what was being claimed so read the linked article.

It's still not 100% clear but elements of it include:

- Historical racist texts (Mein Kampf, 1930s newspapers) which I feel on balance is a good thing to preserve in a library. I would assume having those papers available does more to combat fascism than to encourage it (though who knows really)

- archives of websites that are dodgy (ISIS and Neo-Nazi are mentioned but there must be all sorts of crazy and or bad stuff on the web that gets archived)

- following US law rather than local law (a general conundrum for internet sites like this, especially if you do it for countries whose laws you like but not for other countries)

- providing a place for people to upload and share files anonymously

So yeah, I'm not really a Free Speech absolutist myself (doesn't seem like many that claim they are actually believe it when it comes to things they disagree with) but doesn't feel like they're in the same boat as social media platforms who actively spread bad things if it increases ad impressions. Some similar issues around policing large amounts of content and dealing with different legal, political and moral frameworks at scale though.

TedDoesntTalk11 hours ago

How many books have you read that were published in the 1800s?

I’m betting close to zero.

Unfortunately, most people 200 years from now won’t care about the 70 petabytes the Internet Archive has saved.

Don’t misunderstand: I am glad they do this and love their work. I just think we overestimate the long-term value of this info beyond a very small set of future historians or social historians.

Most people have their lives to live in this moment, and if they have a chance to look backwards before they were born, it’s not a big piece of their time.

wolverine87610 hours ago

Jane Austen, Dickens, Melville, Oscar Wilde, Mark Twain, Tolstoy, Emily Dickenson, Dostoevsky, Emerson, Charlotte Bronte, Lewis Carroll, Victor Hugo, Grimm bros., Goethe, Darwin, Ibsen, Nietzsche, Pushkin ...

It's a pretty good century for literature and other books.

Also, lots of people are interested in history. Those can be best sellers.

Andrew_nenakhov6 hours ago

You must be biased against the French, mentioning only Hugo in your list! What about Dumas, Stendhal, Balzac, Zolya, Flaubert, Maupassant, George Sand, Verlaine, Rimbaud, Valery???!

(That's only from a Russian, pretty sure actual French will add dozens more to this list)

someguy10101011 hours ago

I was reading every newspaper that mentioned spanish flu from the 1910's when covid was starting. I never would have thought that I would have been wishing for easier access to full text search newspaper archives. Here we are :p

instagraham10 hours ago

Quite a spectacularly wrong take on multiple levels.

So much great literature comes from centuries before our own. And considering that the internet is likely to be around forever or as long as humans persist, a snapshot of its initial decades will one day be one of the greatest "archaeological" treasures.

Perhaps your main point is that you do not care for their work nor for the work of literature written before your time. You need not apply your yardstick to anything else in a bid to gauge its value.

SilverRed11 hours ago

How many times have you clicked a link and found the page 404 or redirect you somewhere else? That is the real value of the web archive to me. Wikipedia uses it a lot as well, they automatically snapshot any link used as a reference and if the page goes away they automatically swap the link for a web archive version.

Reading a reference 50 or even 200 years old is not absurd. A post detailing some research findings which is referenced in wikipedia is still greatly valuable. Youtube historians often reference ancient patents to uncover the history of old items.

techbio10 hours ago

Interesting point about managing against link rot.

But what got me is Youtube historians and their "ancient patents..." I want to see the ones for reinventing the wheel.

kart2310 hours ago

I hope you're wrong. Video and pictures coupled with text are a goldmine for me. I would've loved to see a 'vlog' from the 1800s honestly. What was it like for a perfectly normal person, not a career writer? I don't think we have a lot of that, or if we do, it's from a singular viewpoint, and we're subject to their view and biases.

I found a archive of videos in my city from 1970, a street-view like recording of select roads. I pored over it for a couple hours, noting the buildings that were still there, the completely empty hills now filled with houses, etc. That kind of stuff is really cool.

dannyobrien10 hours ago

How many books have you read that were written by people who read books published in the 1800s?

TheCowboy11 hours ago

> How many books have you read that were published in the 1800s? I’m betting close to zero.

Quite a few actually, and I'm not an outlier. Plus there are many adaptations and derivative works that exist.

jfoutz11 hours ago

Eh, Shelly, Dickens, Twain and Wells are all I can think of. Not zero, but a vanishingly small percentage.

mdp20218 hours ago

>I’m betting

You are betting very wrongly. People use large amounts of older literature. Maybe not in your territory - well, be aware then that many cultures do.

>Unfortunately, most people

That the median individual should be considered a parameter is very controversial. (Contextually: services are very easily for interested minorities.)

>overestimate the long-term value

As if Project Gutenberg had not arguably been one of the most important endeavours in history.

>if they have a chance to look backwards

It is a fundamental part of education...

loughnane4 hours ago

Just add a few more that are popular

- Ralph Waldo Emerson

- Henry David Thoreau

- Rudyard Kipling (jungle book)

- Anna Sewell (black beauty) - Walt Whitman (leaves of grass)

- Edgar Allan Poe

- Alexander Dumas (Count of Monte Cristo, musketeers)

- Tocqueville (democracy in America)

kilroy1237 hours ago

I've read several from that time period. There are loads of good books from the 1800s.

dleslie11 hours ago

19th century literature is full of treasures.

Mortiffer10 hours ago

Via librivox for me and many others i consume 1800s content almost daily

5etho9 hours ago

when I read for the first time Balzac's novel - nucingen Bank (1839) [1] and about business 'mindset' at the time I was mesmerized [1] (no wiki for english lang at least no hyperink)