Enforcing expiry of root certificates on a device which no longer gets root certificate store updates is essentially creating “planned obsolescence”. I know it's not the most comfortable thing, but for the sake of avoiding things becoming e-waste too early, it's important that devices allow ignoring root cert expiration.
This is why keeping HTTP open is an accessibility must-have, despite the low risk of ISP MITM and such.
There are millions of devices out there already which are usable for many tasks like reading, searching, and non-critical writing, as long as HTTP is allowed.
HTTPS/SSL introduces many accessibility issues, such as for people with older devices, those who do not know how to set their clock correctly, and many other scenarios.
PKI (public key infrastructure) is much more fragile than people realize. Each browser vendor has its own store of trusted certificates. Java, Node.JS, .NET/Mono have their own stores. These are completely separate from the operating system's store for good and bad reasons. Also, certificate revocations are not even handled like you'd expect: https://www.ssls.com/blog/why-ssl-certificate-revocation-che...
Thank you for sharing this. Despite suffering many issues with older devices and browsers, I never realized it is this bad.
Who decides what’s non-critical? I’d prefer my news articles not be rewritten by the government for example. Catering security to the absolute lowest common denominator (“can’t set a clock”) is not a path to success.
Suppose the likelihood of the government or ISP rewriting my requests is very low, as it is in the U.S.
Suppose also that I am an underprivileged individual whose only device is a 10-year-old tablet I happened to find in the closet.
I'm searching for a crucial piece of information, e.g. how to receive a subsidy benefit or health symptoms.
This is a real-world scenario I have myself witnessed, and have also read about online many times.
Are you REALLY arguing for denying this human being access to the information in the name of "security"?
> Suppose the likelihood of the government or ISP rewriting my requests is very low, as it is in the U.S.
Comcast used to (and maybe still does) inject JavaScript into HTTP requests when users were approaching their transfer caps, so that a warning banner would be shown to let them know they are almost at their terabyte limit.
ISP's have been and continue to be caught playing tricks like this. T-Mobile for instance used to rewrite the playlists that YouTube would send down for HSL to remove the higher bitrates to reduce network traffic.
Just because you are in the US does not mean you are immune from this sort of shenanigans, currently it is used mostly in the name of network management, but it could also be used for nefarious purposes.
I also fully believe this is why public libraries should exist. The library near me has new computers, help that can help the person navigate those sites and even has tablets available for loan.
You act like "no access" can't happen because other things break. If that was the only choice, I'd consider it, but I'm not going to give up everything in tiny incremental bits to maximize access in every edge case.
Also you could disable expiration as a much safer measure.
I don't see how this kind of argument generalizes. Short of armored truck deliveries, cryptographically signed digital transmissions are about the only form of information delivery where it's even possible to get this kind of assurance. Yet civilizations have operated for centuries if not millennia on the premise that we deliver things nonetheless. There is no guarantee at all that the government isn't rewriting the New York Times before it hits newsstands, intercepting and changing television signals, swapping out or reading your mail, injecting mind control serum into foods before they reach the grocery store, or that fluoride isn't used to subdue the population (other than science, but you don't know the government isn't intercepting and rewriting published research). The only assurance you get is all these things are illegal. Governments definitely don't universally follow their own laws, but at some point, the existence of some system of laws is either enough for you, or you go live in the woods with a bunker full of seeds and ammo, or start a revolution to replace the government with a new one.
I decide what's non-critical. What a concept, huh. If I consider an online transaction to be critical, I'll check for https:// in the address bar.
Usually I don't.
Case in point: Firefox 93 now issues gratuitous scary warnings when a .PDF is downloaded over a non-https connection. [1] Right now it only seems to happen if you arrive at the link via a search engine, but it would be silly to pretend they'll stop there. There is nothing OK about this. It literally breaks the whole idea of decentralized Internet protocols.
The obsession with "https everywhere" needs to stop, now. Otherwise, not only will our future landfills groan under the weight of megatons of e-waste that didn't need to stop working when it did, but our collective cultural history online will eventually consist of nothing but undecodable random numbers.
Not everything needs unbreakable encryption. The vast majority of online content doesn't.
> The obsession with "https everywhere" needs to stop, now. Otherwise, not only will our future landfills groan under the weight of megatons of e-waste that didn't need to stop working when it did, but our collective cultural history online will eventually consist of nothing but undecodable random numbers.
If those devices can't even update certs, they absolutely should not be online because they're solid blocks of vulnerable software that will just contribute to botnets.
And the only way for this to contribute to losing history is if you're somehow archiving content by grabbing it off the wire, which seems inefficient anyways.
If those devices can't even update certs, they absolutely should not be online because they're solid blocks of vulnerable software that will just contribute to botnets.
Ah, yes, the presumption of guilt. "You're going to do a bad thing at some point, I just know it. This is probably because you're a moron, while I'm not. Fortunately, I have the solution."
We hear that a lot these days.
ISP MITM is not low risk, and has been already done by Comcast at the very least in the US to inject warnings about bandwidth caps [0]. And with the horrible security of home routers, ISP provided or not, MITM attacks are highly likely on home connections sourced from botnets and malware attacks on routers.
If you had to choose between "ISP may inject a notice" and "no access to critical information at all", which would you go for?
In terms of the topic in this thread, it's not the customers responsibility to deal with expired root certificates. That said, I understand that there is a large issue with devices that drop support way too soon, even though the hardware is good. But the solution is not the weaken the security, but instead to force better standards for how hardware is maintained, and ensuring that there is a long lifetime where either the manufacture supports the device, or the device is completely unlocked and allowed for easy community sourced modifications and updates. That sort of critical information should be secure, because taking an example from elsewhere in this thread, I'd rather be confident I was reading the exact page as intended from the government source regarding how to apply for unemployment benefits and not have to worry that malware in the router is modifying the information on the page to steal information and use it to redirect those unemployment benefits.
And this theoretical attack on home routers is not out of the question at all. How many unmaintained unpatched IoT devices have been abused with malware/botnets. The clock is ticking on mass exploitation of home routers being attacked and it's firmware replaced with one injecting/stealing information from insecure webpages. If the devices can't be updated, we should make sure there are _safe_ alternatives to accessing the information, rather than hoping that no actor is doing things they should not.
I can list so many scenarios with critical information and attacks that could be made if the webpage was not HTTPS with proper certificates. Including school districts in the United States being force to block inappropriate content in order to receive federal funding, and those firewalls abusing non-http content to decide what to block, and school districts abusing that capability to block anything and everything they want. How about a student trying to understand more about LGBTQ+ individuals and the school district inspecting and censoring the exact content inside the pages to remove words like "lesbian" or "gay" because the school considers them "questionable." Or a school blocking articles pertaining to hacking. I have seen that exact last scenario in fact, where certain articles posted here were blocked in my highschool years ago because they were considered hacking and that was apparently not appropriate to view in school. These are real scenarios and not hypotheticals.
For more info on some other various types of fun attacks, see https://www.troyhunt.com/heres-why-your-static-website-needs...
> That said, I understand that there is a large issue with devices that drop support way too soon, even though the hardware is good. But the solution is not the weaken the security, but instead to force better standards for how hardware is maintained, and ensuring that there is a long lifetime where either the manufacture supports the device, or the device is completely unlocked and allowed for easy community sourced modifications and updates.
Yes, that would be nice.
Unfortunately, I have exactly ZERO control over this, and it won't happen for years or decades to come, if at all. And almost certainly not for existing devices.
What I DO have control over is supporting all those devices today, right now, with some MITM risk in certain scenarios.
I've got an old iPad that's still running iOS 10.0.1 (yes, I know). It's used very infrequently and mostly just for reading (PDF documentation, primarily).
Coincidentally, I happened to be in the middle of looking something up right as the clock struck "NotAfter".
My first clue that "something was up" came when clicking a link to another page on one of the sites I had opened... I switched tabs, clicked a link to some other page on that site... nope. Then, I tried visiting a few random sites that are bookmarked. Some loaded just fine, others through errors.
Out of habit, I glanced at the clock -- my ISP's maintenance windows typically start at 2am and they are very good at starting at exactly that time! It wasn't 2am, but it was just past the top of the hour at this point.
I don't recall now if the errors displayed were certificate-related specifically or just "generic" in nature, but I suddenly recalled seeing an article in the past day or two reminding us all of the impending doom and destruction that was about to be unleashed upon us when the root certificate expired.
Using the Firefox browser (which was up-to-date) on my workstation, I tried loading a couple of the sites that were refusing to load on the iPad but I encountered no issues whatsoever -- everything was fine. A quick check of the certificate chain on the "affected" sites confirmed the cause of my problems.
Back on the iPad, I downloaded a copy of the "new" root certificate from my workstation, confirmed, confirmed, confirmed, and was back to whatever it was I was looking into by 10 minutes past the hour.
Somehow I managed to escape all of the Y2K-level atrocities that were supposed to occur and had totally forgotten about the entire "incident" until seeing this here -- and, as you might guess, haven't ran into any other issues related to this since then.
Apocalypse averted (for now), I suppose.
Yes. The procedure you describe could also have been done any time prior to expiry, and more easily of course since the web still worked back then.
The time is completely arbitrary by the way, the people who minted the certificate could of course choose something else, but by default it's likely if it happens to be 14:26:39 UTC when you make the certificate, it's also going to expire at 14:26:39 UTC however many years in the future.
It's a little disappointing that sites catering to people who run old stuff (e.g. both old iPads and old Macs) didn't put much work into e.g. low friction ways to make this happen. Perhaps that's something where Let's Encrypt should have reached out to the right people and made sure this was on their agenda back in the summer.
But perhaps most people in your situation don't pay any attention to such things anyway and would still have been blind-sided.
This won't fix the legacy devices that already exist, but I wonder if there should be some way for root certificates to sign a replacement of themselves. All other signatures would be treated as invalid after the expiration, but this special "this is my replacement" signature would allow non-updated clients to continue to work.
If you can do that, what would be the point of root expiry?
I think that's a valid question
We got bit by the Let's Encrypt root cert expiry. I knew it was coming but based on what I had read and the stats of our users I expected the impact to be slim to none. We used it for some of our support domains, but not the primary ones. Then Sept 30th hit and everything else hit the fan. Thankfully we were able to get another cert provider to issue a replacement within 8 hours but our customer service team hated us for most of that day.
This cert expiration semi-broke my home backup setup. (The backups work fine but the reporting broke, which is subtly more dangerous.)
Duplicati, which is a .NET app, doesn't seem to have the latest certificates when running on Fedora. The same software on Ubuntu works fine.
This thing broke my development and testing environment.
I am in the middle of developing a plugin (https://www.crawlspider.com) that catches SEO changes, certificate changes and errors (crawlspider)
Suddenly the plugin was catching SSL errors for all the domains I was monitoring. Upon checking the domains in the browser everything look fine. There was a green secure icon on all of them.
It was only after several tests and chat with my admin, it became clear that this is a root certificate issue and somehow connected to letsencrypt based certificates.
Out of curiosity, does anybody know how Apple laptops got updated with the replacement for the let's encrypt certificate?
I was surprised to discover that my wife's machine had not automatically updated, several of her websites went inaccessible on September 30th, and I had to do the update manually. I'm not sure what step we missed that and updated certificate wasn't automatically sent to her.
Which OS? For El Capitan for example, no updates were released by Apple.
I will be using those instructions instead to do it manually
https://apple.stackexchange.com/questions/428460/how-to-upda...
Same thing happened to my wife's laptop as well. As far as I know, Apple included the new certs in an OS update.
They did not issue such an update for older versions of macOS (I forget which version of macOS she's running, but its relatively old).
My iMac was still on El Capitan and I updated it to Big Sur to resolve it.
Kinda related to this - anyone tried to fix their xamarin app so that it'd work with letsencrypt? Developers told me xamarin tries to validate both chains so the hack breaks the validation. Not sure if they've come up with a workaround
Possibly related to that - I just fixed my letsencrypt problem on an old Mac by following these instructions [0], which were referenced here [1].
[0] https://docs.certifytheweb.com/docs/kb/kb-202109-letsencrypt...
[1] https://apple.stackexchange.com/questions/428728/why-are-my-...
you can't do that for mobile apps on client devices
Yes. Android <5.1 or s.th. like that failed due to not having chrome available I think. 6.0 and 7.0 seem to have a different issue with the one of the letsencrypt certificate chains.
OIDC for us seems to work on iOS 15, or not, depending on iPadOS 15 , iOS (Simulator), or iOS-out-in-the-wild .
That's a very good workaround on OpenSSL-based systems, although it's kind-of moot now since if you can update the IdenTrust/DST root, you could simply also remove it and add ISRG's roots.
* Once you think about this more though, you realise that it does make sense and that it just doesn't seem logical upon first thought. In order to make this change and it actually have an effect on the client, you need to have access to that client with administrative privileges to change the root store. If your concern is that an attacker might do something like this, well, I'd suggest you have far bigger issues to concern yourself with given that the attacker has admin/root and is on your device!*
Yes, it's true that not just anyone can modify your local trust store (and if someone does, yes, you've likely got bigger issues).
When I started to "think about this more though", my first thoughts were more along the lines of ...
"What assertions have the CAs made -- whether implicitly or explicitly -- regarding their security measures / practices (WRT private key material, of course) once 'NotAfter' has passed?"
... and similarly ...
"What obligations do the CAs have -- either morally, ethically, contractually, and/or in accordance with the Baseline Requirements -- to continue to protect the 'C-I-A' of the private key material after 'NotAfter'?"
I'm fairly familiar with the BR's, having spent more time reading through (portions of) them than I'd have preferred -- and I keep a hard copy here in one of my desk drawers -- I'll happily admit that don't know the answers off the top of my head.
To be clear, I trust [0] that no ("responsible") CA would ever do something like publicly share the private key used to sign a "trusted" root certificate -- or even an intermediate / subordinate, for that matter -- regardless of how much time has passed since 'NotAfter' ...
However, when a CA generates such a certificate (and/or the private key, for the pedants), requests that it be included in one or more of the various root programs, and agrees to the conditions / requirements of such programs -- including, obviously, the BR's -- they're promising that they will keep those certificates (that is, again, the key material) secure. This isn't an "eternal promise", though, and the 'NotAfter' date that they include in the certificate is how they tell everyone how long that promise is valid for -- that is, it is, literally, an expiration date [1].
Once that expiration date arrives, though, their obligation has been fulfilled. I may very well be forgetting some relevant provisions of the BR's at the moment [2] but once the CA's promise has expired, is it appropriate to expect them to continue keeping their promises (i.e., keep trusting the certificate) if they haven't asserted that they will continue to keep those promises (by renewing the certificates)? [3]
--
[0]: Although "expect" (perhaps just "hope") is probably a more fitting word choice here.
[1]: Yes, they can extend -- "renew" -- that promise later, if they so choose.
[2]: And this is the point at which I'd typically go review them to clarify the matter (just for myself) but this entire comment is mostly rhetorical (a "thought exercise", if you will).
[3]: Personally, I think that "trusted" root certificates should automatically become "untrusted" (and, of course, get flagged as such) immediately upon expiration but that's a whole 'nother argument involving operating system vendors, browser makers, and many other parties. Luckily, the operating system and software I use allows me to be the ultimate authority over matters such as these!
>I'm fairly familiar with the BR's, having spent more time reading through (portions of) them than I'd have preferred -- and I keep a hard copy here in one of my desk drawers -- I'll happily admit that don't know the answers off the top of my head.
CAs are required to have a statement on practices for deactivation and destruction of private keys per the BRs; it's 6.2.9 and 6.2.10 respectively.
e.g. for Let's Encrypt: https://letsencrypt.org/documents/isrg-cps-v2.6/#6-2-9-metho...
Right.
Fusing things like this makes sense. If you have a year old Chrome that's still running but is blocked from updating itself, it doesn't do some of its security checks, "Huh," it says to itself, "I am out-of-date" and some checks it would normally require are disabled because it has no way to know what should be checked a year later.
This makes it less safe than a brand-new Chrome. But, duh, of course it is, it also doesn't have the last twelve months of security fixes, meaning there are twelve months worth of publicly known security flaws in your browser.
Android's choice in this makes a lot of sense. If you get root trust store updates, then you stop trusting old roots and trust new ones instead but if you're no longer getting updates then just do the best you can with what you have.