The sum of all knowledge and the sorry state of the web

154 points10
nicbou9 hours ago

It has never been easier to share knowledge, and thus there has never been a greater time to be curious.

Encyclopaedia Britannica is still there, and it's great. But there's also an army of content creators are there to teach you just about anything. If one source doesn't do it, you have more to choose from.

Yesterday, I read an article about North Africa that mentioned in passing Qaddafi's underground river. A few minutes later I was watching a documentary about it, then a variety of videos about that man. My information binge extended to other African dictators, and will probably last a few days longer.

If I'm curious about something, I can go really deep. I'm not constrained to a few paragraphs in whatever book my local library carries.

You could argue that the web is in a sorry state, but if that's the cost of giving everyone, everywhere access to all this knowledge, then it's a deal worth making. This might be more obvious to someone who did not have access to a well-stocked library.

The problem is not availability, but curation. The sum of all knowledge back then was a well-curated book. Now it's literally all of it, unfiltered.

ricardobeat8 hours ago

I’m not sure you and the author are talking about the same thing. He mentions the fact you can’t even read the news from 10 years ago, the content has simply disappeared. No amount of YouTube videos can replace that.

The problem is not “everything, everywhere” or a lack of filters but the extreme commercialization of all content available, closed networks, the short life of URLs…

grumbel6 hours ago

> No amount of YouTube videos can replace that.

The irony here is that Youtube videos from ten years ago are still alive and well. As Youtube makes a much better places for publishing and archiving content than the rest of the Web. With Youtube you don't have to worry about URLs changing or domain names expiring or anything like that. You just publish your video once, get a unique video-id and don't have to worry about anything else. Google's monopoly worked here in our favor for once and they have been reasonably good in not breaking old content (not perfectly, as video annotation got crippled pretty badly).

I think that's where the rest of the Web fell short. The Web has no concept of "publishing". There is no ISBN when you write a blog, no library were you could look up that ISBN. It's all just a file on a server or an entry in a database, that will get mangled and lost in the coming years. Worse yet, the article itself isn't even accessible from the Web, it's mixed together with a user-interface, ads and other stuff or spread across multiple pages. All this makes it quite tricky to keep old content readable and archived for the future.

This also leads to a weird situation that a lot of publications are still avoiding the Web after 20+ years. They publish as ePub or PDFs instead, which you can somewhat access from the Web, but really aren't well integrated into it. But it's by far the easiest way to ensure that a text document published today will remain readable a decade down the line.

londons_explore5 hours ago

Unlisted youtube videos from 10 years ago are all gone... Google made the decision to delete every video 'shared by URL' because of the possibility that the URL generation algorithm had leaked. It was legally less risky to delete all the content than to risk leaking all the content to the open web.

IMO, they made the wrong call - it would have been better for the internet as a whole to notify all users that "We have a new 'share by link' option, and no longer consider the old links private. Please update all old links and then click this button to disable the old URL. If you don't click the button, videos shared by URL may be discovered by others in the future."

VHRanger3 hours ago

> The irony here is that Youtube videos from ten years ago are still alive and well.

They're not, though.

Because Youtube re-encodes videos every couple of years, with new "better" lossy compression algorithms. And each time the videos get successively worse.

Watching a 2008 Youtube video will not only look grainy because it's 360p, but it'll look actively *worse* than it did in 2008 because of all the lossy compression that was applied to it over the decades.

ReactiveJelly3 hours ago

YouTube has definitely kept the original videos for as long as I can remember, so any transcodes you see are only one generation after the original (plus maybe one more for if they didn't launch with this feature)

And yes the new codecs really are better. Same quality at lower bitrate.

_Algernon_5 hours ago

There's plenty of videos on youtube that have been removed. Every few years when I look through my list of liked videos, a couple more are gone forever. Granted, this is likely by the creator themselves, but that doesn't matter when the purpose is archival.

hobs4 hours ago

On a playlist of about 27 thousand videos (me and my cohorts links to each other over ~8 years) I observe about 10% missing, its a lot!

Most of these are not information but amusing, random, short videos, and of course some of it is account deletions/disabling - copyright strikes - and people taking down their own stuff, but still, a lot!

dEnigma5 hours ago

Unfortunately when I look through my old Favourites playlist, which at some point reached the limit of 5000 videos, I can see how many of those videos are now private, deleted, or blocked in my country. The worst part is that in many cases I can't even recover the video title, so I have no idea what has been lost. A possible solution would be to store the titles separately, but I didn't think about this while I added stuff to my collection. I do agree though that the unique, unchanging URL is a huge boon, when I look at the situation in my browser's bookmarks for comparison.

laurex4 hours ago

I made our wedding playlist on YouTube, since I pay for premium, and about 30% is now gone after a couple years. It would be nice if YT at least gave some indication of things they do not plan to delete, i.e. it's an official source not an upload subject to the whims of DMCA.

base6985 hours ago

Noticed this in a playlist of about 80 videos. It's now 55. I have no idea what it is that's missing.

makeitdouble18 minutes ago

> The irony here is that Youtube videos from ten years ago are still alive and well.

Do videos survive account deletion ?

In particular, GDPR has provisions about deleting account info after years of inactivity, and Youtube is apparently not an exception (

So except the chunk of accounts that will stay active for the years to come, a big part of youtube videos should be disappearing progressively.

auggierose5 hours ago

Well, I once had a comment exchange with Sean Young (of Bladerunner fame) on YouTube, but that is gone now. So much for its archival qualities.

jimmaswell14 minutes ago

> you can’t even read the news from 10 years ago, the content has simply disappeared

Isn't most of it on the wayback machine?

mountainb5 hours ago

Not really. There are vast archives of newspaper articles accessible through a web interface. You just need library access. The free, open web is mostly just a spam ocean, but if you make an effort to access the services that catalog useful information, it’s still very useful. The Google web is shit, though.

bbarnett4 hours ago

Yes, old paper newspapers, which are very thin now, and vanishing.

For web based newspapers, good luck trusting longevity there, just because the library has a free interface to an api...

mountainb4 hours ago

There are scans of old newspapers going back ages, and current ones also. Stop using the web, it sucks ass. Use ProQuest and other similar databases. Web searches are wastes of time for most things. Google wants to train you to believe research is not a skill and that you can get usually get good information from the open web. Neither of those points are true.

pasc18788 hours ago

Depends where you read. Also how do you read the news of 50 or 100 years ago - isn't that more difficult.

I use UK news sites. BBC Guardian and Telegraph all have their old articles online

ricardobeat6 hours ago

One of the functions of public libraries is to archive the news. One could browse weekly papers going back decades, either as hard copies or microfilm. There's a good chance major city libraries still do that, or have digital scans.

For example:

bbarnett4 hours ago

Even on Canada (only a few hundred years in the new world), we have microfiche going back hundreds of years in many libraries.

Much has been digitized, I hope they kept the microfiche for longevity.

trinsic23 hours ago

The "All content available" statement is a bit much. I'd go so far as to say there are a good portion of sites that have commercialized information. But I can easily access other forms of information not commercialized by avoiding mainstream views.

davedx5 hours ago

I saw that too, and yet he didn't give any examples of "News that's disappeared". Feels very polemic to me

zasdffaa6 hours ago

From August 30, 1856. Goes back even further but point mmade. Needs a payment to read but it's there.

lordnacho7 hours ago

It's really the curation that needs to be taught to everyone. A big dose of critical thinking skills is what we all need, because in earlier times you could tell the crappy ideas by the way they were packaged: crazy guy at Hyde Park Corner, dude with a megaphone shouting out passages of the bible, crappy home-made flyers. You could work backwards from "nutter is probably wrong" to why he was wrong. Part of why this worked was because it cost something to publish stuff, and so publishers would have a think about what they wanted to spend their resources on.

Nowadays everyone has figured out how to package the message, and it's super cheap to do so and get it out there. (Incidentally, actual packaging is the same now, crappy products used to come in crappy packages, but no more.)

So now to pick apart an argument you have to be a bit more aware of the actual content, and it's a bit harder to get to the bottom of BS.

hobs4 hours ago

Plenty of people either fill the pews or worship on a carpet pointed towards a thing that fell from the sky, I wouldn't give us that much credit.

imiric8 hours ago

> It has never been easier to share knowledge, and thus there has never been a greater time to be curious.

It's also never been easier to share disinformation, and pollute the vast sea of actual knowledge that exists on the internet.

Sources of information are silo'd into proprietary closed off gardens run by large corporations who only serve their shareholders. Searching for information has been corrupted by advertisers and the sheer amount of misleading content, that finding reliable sources often feels like searching for a needle in a haystack.

The one exception is Wikipedia, though it also struggles with keeping factual information, and has its own set of issues.

kevin_thibedeau8 hours ago

Add in the rising tide of ML generated content that is able to get itself well placed in search results, diluting information with real value. It's really irritating to start reading something that starts out seeming legit and then you get in a few sentences or paragraphs before it becomes incoherent nonsense.

morpheos1376 hours ago

Wikipedia can be show to be biased or unbalanced about political issues. Since wikipedia views newspapers as reliable sources their articles are often skewed by whatever is the conventions of the day.

denton-scratch4 hours ago

I second the call that WP is web done right. Yes, of course there's bias; it's not possible to produce bias-free content, and WP's particularly bad in the fields of politics and history, and really any field where facts aren't settled and feelings are strong.

Enter critical thinking. If you dig just a little (e.g. read the talk pages and the edit histories), you can soon learn that some topic has been taken over by POV-pushers and is unreliable. Anything to do with Israel/Palestine/West Bank is unreliable; the boss is a zionist, and so are a lot of the senior staff, so it's not surprising. But WP is a million times better than the web that search engines expose.

Incidentally, I usually search using DDG. But DDG seems to hate Wikipedia; WP results usually don't show up on DDG until page 2. Google surfaces WP results on page 1, if not at the top of the resultset.

_Algernon_6 hours ago

It may never have been easier to share knowledge but it also hasn't ever been easier to share misinformation.

It also hasn't gotten easier to find information. Search results are worse now than 10 years ago because of Search Engine "Optimization". Google search needs uBlocklist[1] with a steadily growing list of manually curated domains just to poorly approximate usability.

Additionally, sites load slower every day. Why does every partial refresh of a site require 2 seconds, even though I'm on a decent computer? On phones sites are practically unusable.

The benefits you're touting for the web could be accomplished with Web 1.0 level of tech or even simpler protocols such as Gopher[2] or Gemini[3]. Everything else is an overall decrease in accessibility, usability, user experience, and the ease of finding or sharing knowledge.




bombcar3 hours ago

I find that you used to find “silos” of information when searching - if you were looking for information on toilets say, you might end up on a love site [687] dedicated to plumbing, which would have a whole cornucopia of information and knowledgeable people.

Now you’re less likely do find that kind of thing and more likely to find a video or a SEO optimized site - which can be much more difficult to parse for verifiability.


gsatic7 hours ago

Curation won't solve the issues of the Knowledge Gap Hypothesis or the Information Defecit Model or the Digital Divide or Info Asymmetry.

If we test people on where they went "deep" there is a good chance most will fail the test.

We don't even know what we are trying to do with these networks.

mudrockbestgirl9 hours ago

IMO the worst side effect of current web of knowledge is what I'd call the illusion of knowledge. When it was more difficult to access and publish information, that imposed a much higher bar on what was being consumed. These days, people watch a 10-minute YouTube video or read a reddit comment or twitter thread and believe (perhaps unconsciously) that makes them knowledgeable in said topic. They will then, in an absolutely confident tone, display their expertise by answering questions and stating their opinion as if it was a fact. More people read this, and the cycle begins.

You see it all the time on HN and other forums. If you're an actual expert in a specific (usually scientific) subfield and you read comments about an article in that field, you find that a large percentage are not just factually wrong, but also written in an extremely confident tone by people who have probably studied the topic for about 10 minutes.

By having easy access to all this information people have stopped being humble about what they don't know.

Tenoke8 hours ago

People also used to be confidently wrong all the time before the internet was ubiquitous, except then it was hard to quickly verify they were.

denton-scratch4 hours ago

This is true.

Long before the web, when I was a child, my father would chide me for my bold claims unsupported by facts with the phrase "Confident; but wrong."

mudrockbestgirl7 hours ago

True, but a difference is that they could not spread their confidently wrong opinions globally, only locally, and that opinions were tied to their identity.

Take my mom for example. She's 80 years old and doesn't use the internet that much. She is confidently wrong about a lot of things she sees on TV or hears on the radio. A recent example is COVID misinformation.

The difference is that my mom can't easily influence millions of others because she doesn't have the reach, but also because people are unlikely to take the word of an 80-year old person without any medical credentials or training seriously. It's much easier to look "legitimate" when you are hiding behind an online persona. If my mom wrote a blog or posted on HN/reddit, she could certainly come off as a doctor, or even lie about being one, and many would believe her. Doing this locally, in person, is much harder and riskier.

marcus_holmes4 hours ago

> True, but a difference is that they could not spread their confidently wrong opinions globally, only locally, and that opinions were tied to their identity.

> If my mom wrote a blog or posted on HN/reddit, she could certainly come off as a doctor, or even lie about being one, and many would believe her.

I don't think your mom (likeable as she no doubt is) could get an audience of millions just by posting her opinion on HN/Reddit/Social media.

I think the situation is pretty much back to what it was: most of the population have a limited reach of influence. Some people have a much greater reach.

The difference is that the people with greater reach used to be trained journalists who held to a code of conduct and were given that reach by institutions. Now there is no such code of conduct, and the assignment of audience reach is more random, and totally uncontrolled by any institution.

j-bos5 hours ago

Pre internet, people could not spread confidently wrong opions globally, but connected and well connected people could. Take a look at the wave after wave of popular, and misleading non fiction books printed last century, or the "scientific" food pyramid, or the testimony of nurse Nariya. On and on it goes, as it has, only now, anyone can play.

lmm4 hours ago

Many true things about COVID were labelled "misinformation" at one point or another.

The publishing gatekeepers of a few decades ago projected an image of confidence, but I'm not sure they were actually any more accurate than random youtube videos. The media establishment of today certainly doesn't seem to be.

tomrod1 hour ago

That's a mislabeling. True facts were batched together with false conclusions, and should have rather been classified as disinformation since it was performed by people who knew better in many cases.

sandruso8 hours ago

Its good to remind myself this . Not once I thought I know something after watching a summary video. When I tried to explain the topic to somebody else I struggled. If you can’t explain you know nothing. Simple as that.

denton-scratch4 hours ago

> If you can’t explain you know nothing.

Strongly agree. And I'll raise you: if you can't explain in simple, plain language that a 12-year-old could understand, then you are not enlightened.

I concede that some subjects are intrinsically complex; e.g. the cosmological history of the Universe. But a large part of the reason that topic is complex is because it's not settled; we haven't got to the bottom of it, so there are loads of unanswered questions. How can you explain cosmological inflation in language a 12-year-old could understand? Well, I can't explain it to myself, so I sure as hell can't explain it to a 12-year-old.

wslh2 hours ago

At the same time you see people (mainly young people) learning at a faster pace watching videos.

danrocks1 hour ago

[Citation Needed]

sorisos7 hours ago

I also believe it is a trend not to go over the head of consumers. Can be seen in older documentaries and political discussion. Perhaps this is just a result of everyone already feeling like a expert and should not be insulted by complex language etc.

trinsic23 hours ago

Id say it's gone the other way too. Expert opinion echo chambers are rampant on the internet preventing paradigm shifts in science.

muspimerol8 hours ago

I wonder if this is just an inevitable outcome of the majority of society being online. The beginnings of the internet are rooted in academia and hobbyists. Early adopters were experts in their respective fields with analytical minds. Now that everyone is online, perhaps the average user just better reflects the average member of society. In other words, maybe it's not the content of the internet, it's the users.

agumonkey8 hours ago

I agree. There's a healthy factor form entry cost. There's also an healthy inertia into asking more than short term investment from your mind.

Map is not territory and I say this after believing it far too long.

It's a big fallacy behind the information highway roots of internet.

And that's half of it.

morpheos1376 hours ago

What is the most bizarre thing to me about the online zeitgeist is how so many people will allow comments of anonymous or pseudonymous strangers on places like Reddit or Twitter to shape their world view. Including journalists. An extreme or inaccurate view may start on social media and be normalised through repetition on social media and subsequent validation by main stream journalist.

In the case of Reddit in particular what is it that gets people to trust anonymous strangers? It is bizarre and seems like a mind virus. If an anonymous stranger tells you what you want to hear then you are apt to ingest it uncritically.

For example on social media there is a notion that nuclear brinksmanship with Russia over Crimea is acceptable.

gatonegro28 minutes ago

> what is it that gets people to trust anonymous strangers? It is bizarre and seems like a mind virus. If an anonymous stranger tells you what you want to hear then you are apt to ingest it uncritically.

I guess it has something to do with lack of trust in Established Sources of Information—TV, newspapers, experts and other various figures of authority. The biases of the Established Sources has become more apparent over the years, to the point where scepticism, doubt, and even defensive cynicism are fairly common default attitudes when dealing with the information they provide.

The trust in anonymous strangers is, at least in part, the result of them not being Established Sources. It's not a lying politician, or a deceitful news anchor or journalist, it's just another well-meaning regular person like you. That alone makes you more receptive to their message. If their message happens to align with your existing beliefs, even better. Of course, it can get into cultish/conspiracy territory if any of those beliefs are directly opposed to the mainstream narrative.

null_object6 hours ago

> What is the most bizarre thing to me about the online zeitgeist is how so many people will allow comments of anonymous or pseudonymous strangers on places like Reddit or Twitter to shape their world view. Including journalists.

I totally second this, and have a recent concrete example where at the beginning of this year Sweden's (probably) most serious newspaper (what I'd consider a 'journal of record' whose articles should be a point of historical reference), published a long-form retrospective article comparing Sweden's handling of Covid with the way other countries had handled the epidemic, and included several internet 'myths' that had bandied-around on social media, citing them as facts, and even including one 'interview' with what purported to be an eyewitness of one event, which turned-out to be taken from a Facebook post.

I wrote and complained to the responsible editor with citations showing how and where the article was wrong, and a few very grudging emendations were made (effectively saying that even though the reports were still probably true, they couldn't be 'verified').

Totally horrified me that, in wanting something to fit their facts, journalists simply accepted fiction they read on Facebook and regurgitated it in their articles.

I used to hold them in greater regard than that.

base6985 hours ago

Lack of solid real world social networks. Religion used to be a solid defense but people are now less religious.

gnz115 hours ago

Swap out HN for Reddit and your comment still stands. This place isn’t really all that different.

trasz6 hours ago

>When it was more difficult to access and publish information, that imposed a much higher bar on what was being consumed.

And yet fringe theories, from quack medicine to religions, weren’t any less widespread than now. They were just fewer of them.

EDIT: Now I think about it, they were more prevalent than now. Look at homophobia - it got global, affected everyone, even non-Christian cultures like China, for centuries.

tgv5 hours ago

> religion ... homophobia

Do you really consider that knowledge?

> homophobia - it got global, affected everyone, even non-Christian cultures like China

Do you think that aversion of homosexuality starts and ends with Christianity?

trasz4 hours ago

Homophobia generally spread throughout the world with Christianity, carried by colonialism. Again, China is a good example. And fixing homophobia in western societies strongly corresponds to decreasing importance of religion.

It’s not limited to homophobia of course - pretty much every single Catholic claim about human sexuality is antiscientific bull - but I think homophobia is a good enough example of a harmful, false belief that got more popular than anything post-internet.

tgv1 hour ago

For one thing, you may be underselling the homophobia in Islam a bit. And there's Tacitus, who says that the Germans punished homosexuals. That's a bit before the Christians, and it comes from someone from a culture not opposed to the practice.

Second, I find it hard to believe that the Chinese were turned homophobic, and remained so, by a few Christians despite millenia of tolerance, when the vast majority of the country isn't even Christian. It smells like bad historicism.

goodpoint7 hours ago

> They will then, in an absolutely confident tone, display their expertise by answering questions and stating their opinion as if it was a fact.

The irony of talking about this on HN, the home of Dunning-Kruger effect.

Eleison238 hours ago

Remember how Commander Data would scan through a few million files during a 10-second montage, or how Keanu jacked in briefly, woke up and exclaimed "I know Kung Fu!"

Yeah, that's exactly how it works today.

BrainVirus34 minutes ago

>Ever tried to look up some news from 12 years ago?

I have a better one for you. Ever wondered why it's so hard? Why web protocols have nothing related to archiving? Why web browsers are a hellscape for aggregating information over time in a meaningful way? Why this continues to be true, despite countless Microsoft and Google engineers writing all these heartfelt posts about knowledge?

If your answer is "because it's hard to implement" than you understand nothing.

doliveira26 minutes ago

From my admittedly limited understanding, the failure of the semantic web is one of mankind's biggest missed opportunities. Now the knowledge graph is just locked behind Google's neural network layers and only being used for ads.

stjohnswarts24 minutes ago

The idea behind semantic web was inspiring and great, however it required considerable work on the part of people creating stuff for the web and that was never going to happen. Maybe it could have happened in some things like academia based or knowledge based websites, but on the larger scale it was doomed.

vmoore50 minutes ago

Just going to leave this gem of a video[0] here. Whilst I agree the web has been walled gardened into various silos like social media, and people think Facebook and Twitter are the Internet, it's still read/write. The blogosphere is still ticking along nicely and last time I checked it's thriving.

Yes, people have gamified Google and search to get traffic and the blogosphere of old has largely been co-opted by profiteering gluttons, but there's still hope. Surf Hackernews enough and you'll find little gem posts that don't have an ulterior motive and are not 'monetizing' their content and sprinkling it with affiliate links and ADs. They just want to vent, exchange techne, and share knowledge.

Then there's Wikipedia which has remained AD-free for as long as I can remember (apart from their donation banners which I don't mind). Wikipedia is the coolest thing ever and my IQ has probably gone up a few notches over the years because of it. It is the closest thing to getting home-schooled without going through formal education, and you can verify all the claims made in its entries by going to the footer section and reading various citations usually written by esteemed scholars.

The web is in a sorry state due to the commercialization and walled garden silos, and also because of the proliferation of smartphones which are mere consumption devices IMHO and not designed for producing any meaningful or substantial content, apart from maybe uploading photos/videos to Instagram or writing tweets etc I can't write a blogpost on a phone because I have bad dexterity, and I typically have to have 100+ tabs open to verify claims, provide sources, do cross-referencing, find relevant images etc...all something done best on a workstation PC or laptop.

Some context: I have professionally blogged for more than five years but due to reasons I won't go into here, I have stopped. I'm thinking of jumping back in, only this time armed with the wisdom of my previous blogging shenanigans. Failure is an opportunity to start again more intelligently!


AshleysBrain5 hours ago

Blogs are still an underrated goldmine of knowledge, especially in tech. I find academic papers often too abstract or opaque, textbooks are good but generalized, and documentation is reference-like. Stumbling across a tech blog where someone explains some fairly specific and difficult problem they had, and an interesting solution they found, can be exactly what you needed to solve a problem.

The web has its problems for sure, but don't forget there are still gems out there which the web has made possible. I'd love to see a comeback of blogging culture, but I guess a lot of that has been sucked in to social networks now.

bombcar3 hours ago

Blogs are wonderful and still there - they’re just much harder to find because there is so much OTHER content now. The web used to be almost nothing but blogs.

For example, this blog is pertinent to what I’m doing and I didn’t find it for weeks: and only found it by a link to a YouTube video from another one from another one.

saperyton2 hours ago

A great resource to find new blogs is the Thinking About Things newsletter. Been getting it for a while and it's a great way to find new blogs to read.

llaolleh2 hours ago

Part of why the web is shit is low barrier to entry. More often than not, if you read classics or older books, it's information dense. Every prose and sentence constructed had some economy baked in.

Now everyone and their mom, as well as bots and marketers,can spam right on over.

We either need a search portal with aligned incentives, or perhaps a new internet with none of this crap.

MMS211 hour ago

Check out Gopher/ Gemini protocol

masa3316 hours ago

I don't get how the author can value all the things in the article and yet work at Microsoft on a bloatware like Edge which is not possible to remove, is pushed hard by Windows against other browsers and used as a part of a giant ads engine in itself

_gabe_3 hours ago

> and I now work on the browser that comes out of the box with any Windows machine (working on a Mac most of the time).

Is this why Microsoft products are getting progressively worse every year? How can you work on a product and then never use it natively and expect other people to enjoy using it? Or worse, like with windows 11, it leads to the product morphing into something the users never wanted. Because developers want to conform it to what they're used to using. I don't know, it kind of baffles me the way most developers view the products they make.

mellavora9 hours ago

Your father's books are real treasures. Were it my library, they would have a place of honor.

I want to put your post in the context of yesterday's HN front-page post lambasting the EU for trying to build a better search engine. The bulk of the comments suggested that a government effort could never be as good as a commercial effort.

Your post is a strong counter-argument. All of the points you mention on the massive decline in quality of web content are due to the web being driven by commercial efforts.

Likewise, comments on this current HN front-page post "Despite faster broadband every year, web pages don't load any faster" also seem to explain the poor state of the web as being due to commercialization of everything (even the comments about the need for cookie banners-- the logic behind the cookie consent is to regulate the commercial collection of user data).

Google started as a DARPA project, and was a great engine while it stayed true to that ethos. It was the need to commercialize it, thus setting perverse business incentives, which has destroyed it.

Your post praises libraries. Which are seldom commercial ventures.

The economics are simple. I don't know why this even continues to be a debate on HN.

- A socially created entity (corporation, government, "charitable" organization, ...) needs money to function.

- Money comes from capturing a portion of the value created by the organization.

- The most efficient organizational structure depends on the relationship between value creation and value capture.

Thus: if the product/service generates immediate and focused value, the value capture can be directly linked to the product and a business is optimal. Think: a hamburger.

If the product/service generates long-term and diffuse value, the value capture also needs to be diffused, i.e. taxes. Thus a government. Think: the road network which allows the raw materials and the customers to get to the hamburger store.

I leave the case for charitable orgs as an exercise for the reader :)

disclaimer: strongly pro-business, have founded 2 personally, assisted several others.

Eleison238 hours ago

>Google started as a DARPA project, and was a great engine while it stayed true to that ethos. It was the need to commercialize it, thus setting perverse business incentives, which has destroyed it.

Google started as a what now? This is an interesting thing to say during a discussion of access to information. I'd be happy to read your explanation of Google's "DARPA roots", and your citations to sources explaining how that came about and how they were "destroyed" when they "strayed" from DARPA... that should be a fascinating read.

mellavora7 hours ago

It is indeed a fascinating read. Here you go!

>A second grant—the DARPA-NSF grant most closely associated with Google’s origin—was part of a coordinated effort to build a massive digital library using the internet as its backbone. Both grants funded research by two graduate students who were making rapid advances in web-page ranking, as well as tracking (and making sense of) user queries: future Google cofounders Sergey Brin and Larry Page.

>The research by Brin and Page under these grants became the heart of Google: people using search functions to find precisely what they wanted inside a very large data set.

mellavora7 hours ago

You also asked 'how they were "destroyed"'

I refer you to some interesting discussion here:

Google Search is Dying 1561 points

Every Google result now looks like an ad 972 commments

Google no longer producing high quality search results in significant categories 1275 comments

talideon7 hours ago

Uh, DARPA had nothing to do with the founding of Google. Brin and Page had NSF funding though Brin's graduate fellowship and the Digital Library Initiative, and that was before the official founding of the company.


mellavora7 hours ago

You may (or may not) be right about DARPA, but you assert they were government funded.

And my main point is that government (funding) is the economically optimal approach for services which produce diffuse value.

See also the comment from marginella_nu in the post I linked to. Marginella is building a fantastic alternative search engine Their view: "Arguably the biggest most unsolved problem in search is how to make a profit"

i.e., capturing the value produced.

kagi attempts to do this with a paid tier. I hope it works for them, great product and really responsive team.

Bing tries to capture value by collecting the Microsoft tax; not exactly government-level, but on those lines.

closedloop1292 hours ago

The social web is no problem. We can choose what we visit like the author's father chose to buy those books.

What is missing is a beacon of light in the desert of choice. There is no lack of knowledge anymore but a lack of orientation. The author's father knew that he could buy knowledge in a bookstore. Where do we go to find orientation on the web?

People don't know. That's why they are stuck in the social net with its memes.

seydor3 hours ago

> The web we have these days is in a sorry state

The web is one of humanities greatest inventions, right up there with Gutenberg or even more. It's a shame that people focus on politics and miss the rest of the web. There are two webs, one is a medium of mass manipulation, just like all media before it, the other is the library of all human knowledge, ever, everywhere.

The printing press precipitated major societal changes, but the internet has yet to. We still follow the rules set 400 years ago, voting some humans to rule us every 4 years, and have not questioned the system under the current situation , where everyone can create and access information everywhere, instantly. The internet will take us to new politics , we just have to invent it.

trinsic24 hours ago

I appreciate your personal account about where society is heading related to the Internet.

I avoid for-profit social networking websites because for the free flow of information because I realize that these sites only represent a small portion of what the web is about. I know the Internet is really best when I read from people that self-publish. I also publish articles on interpersonal work and the state of technology myself. As a "principal product manager in Microsoft working on tooling to enable people to do more on the web." I wonder what you think about Microsoft and Apple creating walled gardens in there respective OS's? I recently switched to Linux and [wrote]( about why I think the biggest threat to the free flowing information of the net has to do with how we allow our technology that connects to the net to become restricted.

mikewarot3 hours ago

If we all had the Memex that Vannevar Bush proposed[1,1*], many of the losses we all discuss in these threads may have been avoided. We now have massive local storage, and should be able to freely share data by hosting our own stuff on our own machines. We could have done what the editor of the magazine implored us to do:

"As Director of the Office of Scientific Research and Development, Dr. Vannevar Bush has coordinated the activities of some six thousand leading American scientists in the application of science to warfare. In this significant article he holds up an incentive for scientists when the fighting has ceased. He urges that men of science should then turn to the massive task of making more accessible our bewildering store of knowledge. For years inventions have extended man's physical powers rather than the powers of his mind. Trip hammers that multiply the fists, microscopes that sharpen the eye, and engines of destruction and detection are new results, but not the end results, of modern science. Now, says Dr. Bush, instruments are at hand which, if properly developed, will give man access to and command over the inherited knowledge of the ages. The perfection of these pacific instruments should be the first objective of our scientists as they emerge from their war work. Like Emerson's famous address of 1837 on "The American Scholar," this paper by Dr. Bush calls for a new relationship between thinking man and the sum of our knowledge. — THE EDITOR"

We don't have that, and it makes me sad. We should fix this inadequacy, but there are now so many interests in the entrenched model that I think they would squash something that freely allows copying like a bug.



[edits - quoted editor's introduction, revised/extended]

arcbyte3 hours ago

Is it ironic that I can read the article you posted because it's behind a paywall?

mikewarot2 hours ago

I was lazy... thanks for prodding me to do better.

carvking7 hours ago

Interesting article until: "Use to check if the shocking thing you just read is real."

Mind baffling that this article could end with this.

denton-scratch3 hours ago

Well, if you read something surprising, then you should probably check it somewhere, unless it's trivial. Snopes might be a reasonable first check.

carvking24 minutes ago

Who fact checks the fact checkers ?

Once you read something totally unreasonable in a fact check, why should you go there as a source of truth anymore?

Wikipedia is probably a better bet - at least you have history of edits and sources.

marcinzm7 hours ago

>Ever tried to look up some news from 12 years ago?

Internet Archive which is a lot easier to search in than going manually through microfilm.

izacus7 hours ago

Except that people on this very sight are fighting very very hard to ban scraping from sites like Internet Archive and keep propping up IP laws that make such libraries illegal.

johnchristopher6 hours ago

I love old books like this !

I managed to salvage this one when my grand parents went away

I don't think it's worth much but it's old and smells nice and feels good to handle.

cm21877 hours ago

On the other hand the digital assets that people make a personal copy of never age, so chances are higher that we will find some old writing that we thought was lost when the server shut down.

Of course that's in a world where we still have personal computers, not apple style locked down devices.

amadeuspagel5 hours ago

Tells a touching story and how that motivated the author to work on browsers, standards and CMS. Wonderful. But the rest is awful. Whining about ads and paywalls without suggesting any better way for websites to make money. Telling people to check after they read something. Well, I have a better idea: Just don't read anything other then snopes in the first place. If you accept snopes as the ultimate arbiter of human knowledge, there's no point in poisoning your brain with anything else.

abitnegative4 hours ago
eimrine9 hours ago

I use to ask people about the thing exposed in your photo (why the seasons change, what the moon phases means) to make a quick test about random interlocutor's intelligence to see is it worth to keep discussion. Your father passed this test in absentia.

istinetz9 hours ago

nowadays, you can just ask about astrology. It's both obviously wrong and socially acceptable to believe in. It filters a solid 30% of the population, at least in my country.

Lammy9 hours ago

I dunno, I'm willing to believe the timing of birth can influence a child's development since e.g. the ecological seasons will influence the amount and types of socialization / experiences available to them at a given point in their development. The stars or planets probably have nothing to do with it, though, besides being a way to articulate "this kid said their first word during winter" or whatever.

Plus I think it's nice to avoid falling into "I don't hate women; I just hate everything women like" lol

swilliamsio48 minutes ago

The timing of birth does at least slightly influence a child's development through the relative age effect - the oldest kids in each academic year get a few benefits over their peers.

hcks7 hours ago

Your explanation is 100 times worse than basic astrology to me. It's astrology wrapped in scientism.

We can't even be sure about what part of character comes from all of the environment versus what part comes from genetics without very careful experimental settings.

Obviously any systematic change due to ecological seasons would have a minuscule effect, drowned in the noise of a million other parameters.

It's not even like astrology can make any predictions about anything, there is nothing to be explained.

vidaj8 hours ago

Seems like something that could be "easy" to validate. Compare children's development in the US with those in Australia and see if there's a 6 month difference.

Note I put easy in quotes, because this should be provable, but I don't think it's trivial to do so due to all the different factors that contribute to a child's development.

Edit: And by "6 month difference", I mean that if seasons have anything to do with child development, you would see the same kind of development in children born in March in the US and children born in September in Australia.

bee_rider8 hours ago

Historically, I bet you could find trends based on which seasons certain developmental landmarks were hit. I'm sure it isn't a coincidence that Leos are attributed all the heroic traits, when they are mostly gestating in the Summer, and get their first couple months of life in the Fall harvest season.

Nowadays I guess it would line up to school seasons. If you are born in August your parents can arrange for you to be among the oldest kids in the class. First couple grades, that extra year of development helps out, and then you get the reputation as the smart kid, it snowballs...

Baseball player? Bet he's a Leo.

denton-scratch2 hours ago
istinetz6 hours ago

>I dunno, I'm willing to believe the timing of birth can influence a child's development

That's a post-hoc justification for an obviously ridiculous belief.

1) Astrology makes much stronger claims. "Today, you will be lucky in love, but avoid conflicts at work". You're committing a motte-and-bailey.

2) If that's the justification, say "winter babies" and "summer babies", not virgo and capricorn.

3) If that's the justification, there should be much stronger effects, such as "born in an ex-communist state, so different nutrition", "born in colder climate, so less sun" that absolutely dwarf that effect. Yet you don't see idiots making intricate psychological profiles based on what latitude you were born in.

>I don't hate women; I just hate everything women like

It's a retarded belief regardless of who holds it.

>There are many reasons that the gender gap in modern astrology belief exists, but some sociologists say that it is, like most things, a result of the patriarchy.

Well, this is something else people say that makes me immediately lose respect for them.

emptyfile8 hours ago

I'm so infuriated by your link I'm at a loss for words.

EDIT: This is an UNIVERSITY newspaper? I thought it would be some kind of british tabloid or something... depressing

Peritract7 hours ago
eimrine9 hours ago

Too obvious if an interlocutor is an engineer. Too severe if an interlocutor is a woman.

dazc9 hours ago

Typical Virgo

zaknil4 hours ago

If that feels too intrusive, just ask people if they use to check if the shocking thing they just read is real.

eimrine4 hours ago

Cursive means irony? I visited the website but I have not seen any good use for me in this website.

tiagod5 hours ago

Knowledge does not imply intelligence. Nor is the opposite true.

eimrine5 hours ago

Since Astronomy is considered one of seven Liberal Arts, knowing the very bases of orbital mechanics is not about knowing for example difference between interfaces and abstract classes. Understanding this astronomical subject speaks to a general curiosity that in my opinion correlates with intelligence.

An interesting observation, this is not the first time I've talked about this little test on HN, but this time it was not upvoted for.