Timeline of the xz open source attack

It would be intetesting if Lasse Collin published his off-list interactions with 'Jia Tan' and any of the other pseudonyms, to get an even better angle on the social engineering parts. Apparently a large part of the campaign was via private channels to Lasse.

I wondered about this. I saw the note at the bottom "RedHat announces that the backdoored xz shipped in Fedora Rawhide and Fedora Linux 40 beta" but saw nothing in the timeline explaining when/how it made it into Fedora.

That's a great first step, but ready your pitchforks for this next take, because the next step is to completely eliminate Turing-complete languages and arbitrary I/O access from standard build systems. 99.9% of all projects have the capability to be built with trivial declarative rulesets.

This doesn't read as a technical failure to me. This was 99% social engineering. I understand that the build system was used a vector, but eliminating that vector doesn't mean all doors are closed. The attacker took advantage of someone having trouble.

While I do agree that M4 is not great I don't believe any alternative would have prevented this attack: you could try translating that build-to-host file to say python with all the evals and shell oneliners and it would still not be immediately obvious for a distro package maintainer doing a casual review after work: for them it would look just like your average weekend project hacky code. Even if you also translated the oneliners it wouldn't be immediately obvious if you were not suspicious. My point is, you could write similarly obfuscated code in any language.

What's inscrutable code? Was it m4 or sh or the combination of the two?

Who will pay for all the rewriting you want done? Or even just for the new frameworks that are "scrutable"? How do we guarantee that the result is not inscrutable to you or others?

There is so much knee-jerking in this xz debacle.

(And I say this / ask these questions with no love for autoconf/m4/sh.)

I do think we have enough eyeballs at this point that we should stop entertaining the Dancing Bear in low level libraries and start insisting on crisp, self-explaining code. There are a lot of optimizations pushed into compilers these days, and there are a lot of architectural changes that can make things fast without making them inscrutable.

We should be moving from No Obvious Bugs to Obviously No Bugs (Tony Hoare).

That would be an improvement for sure... but this is not fundamentally a technical problem.

Reading the timeline, the root cause is pure social engineering. The technical details could be swapped out with anything. Sure there are aspects unique to xz that were exploited such as the test directory full of binaries, but that's just because they happened to be the easiest target to implement their changes in an obfuscated way, that misses the point - once an attacker has gained maintainer-ship you are basically screwed - because they will find a way to insert something, somehow, eventually, no matter how many easy targets for obfuscation are removed.

Is the real problem here in handing over substantial trust to an anonymous contributor? If a person has more to lose than maintainership then would this have happened?

That someone can weave their way into a project under a pseudonym and eventually gain maintainership without ever risking their real reputation seems to set up quite a risk free target for attackers.

Autotools-based build infra is always crufty, fragile, slow, and fugly. In a lot of cases, it sucks because it's what we have right now without doing the work to replace it with something equally flexible.

Build infrastructure should be minimal, standardized, and not subject to endless special, undocumented fragility.

cmake, just, conan, meson, and bazel (and forks) exist.

But I've still yet to see a proper build system that does feature detection in parallel and concurrently, or supports live, incremental, continuous, cached rebuilding.

Functionality like IFUNC is completely inexcusable outside of dev/debug builds imo. It's rot.

That and we need to pay open source maintainers and find new ways to support them.

And all code that gets linked into security critical applications/libraries needs to be covered by under some sort of security focused code review.

So no patching the compression code that openSSL links to with random junk distribution maintainers.

That's an improvement, but ultimately not enough, actually. The article hits at this, and you should definitely read it:

Being able to send xz patches to the Linux kernel would have been a nice point of leverage for Jia Tan's future work. We're not at trusting trust [1] levels yet, but it would be one step closer.

[1]: https://research.swtch.com/nih

With all the CI tooling and containerization, it seems to be going in the opposite direction.

[flagged]

Great, let the cmake dummies wander off into their own little dreamworld, and keep the professionals dealing with the core stuff.

> Far too many people, including developers, seem to just accept upgrades as always-good instead of carefully considering the risks and benefits.

Another example of this was log4j: if you were still using the old 1.x log4j versions, you wouldn't have been vulnerable to the log4shell vulnerability, since it was introduced early in the 2.x series. The old 1.x log4j versions had other known vulnerabilities, but only if you were using less common appenders or an uncommon server mode or a built-in GUI log viewer (!); the most common use of log4j (logging into a local file) was not exposed to any of these, and in fact, you could remove the vulnerable classes and still have a functional log4j setup (see for instance https://www.petefreitag.com/blog/log4j-1x-mitigation/ which I just found on a quick web search).

Did log4shell (and a later vulnerability which could only be exploited if you were using Java 9 or later, because it depended on a new method which was introduced on Java 9) lead people to question whether always being on the "latest and greatest" was a good thing? No, AFAIK the opposite happened: people started to push even harder to keep everything on the latest release, "so that when another vulnerability happens, upgrading to a fixed version (which is assumed to be based on the latest release) will be easy".

About a decade ago, the industry shifted from slow, carefully evaluated infrequent updates (except for security) to frequent, almost daily updates that are mandatory. I'd say this was pioneered by the Chromium team and proved to be beneficial. The rest of the industry followed.

Now we're in a position where most projects update so quickly, that you don't really have a choice. If you need to update one component, there's a good chance it will require that many more as dependencies in your project will require an update to be compatible.

The industry as a whole sacrificed stability and some aspects of security for faster advancements and features. Overall I'd say the net benefit is positive, but its times like these that remind us that perhaps we need to slow things down just a little and do a bit of a course correction to bring things into a better balance.

All things considered I'm not sure that'd be such a good thing

How many security issues spring from outdated packages vs packages updated too hastily?

This drove me crazy at previous companies where our build system dependencies were specified with wildcards so they would just auto pull in the latest version whenever you built the project. Not only are there security issues with that, as seen here, but it destroys your ability to have deterministic builds; every time you build the project it might be building against a different set of dependencies. You lose the ability to even compare builds against each other, because even though the builds are for the same exact project code, the dependency binaries might be different, and so the two builds could run differently.

The 2017 WannaCry ransomware attack would be a good counter example, the virus spread even though it was already fixed in an update from MS a bit more than a month before.

I'm kinda the opposite. Way too many times I've seen "upgrades" actively remove things I liked and add things I hate. I hold off on letting mobile apps update because they almost always get worse, not better.

Wouldn't that just result in exploits written for old versions? A successful exploit for something that everyone is running might be worse, than a backdoor on blending edge systems.

Everyone being on different versions results in something like a moving target

well, it's the bazaar vs the cathedral, isn't it? bazaar moves a lot faster. Everyone likes that part, except when it breaks things, and when they have to chase an upstream that's constantly churning, etc. but most people don't consider that a cathedral itself might have some engineering merit too. cathedrals are beautiful and polished and stable.

I highly encourage people to try freeBSD sometime. Give ports a try (although the modern sense is that poudrie is better even if you want custom-built packages). See how nicely everything works. All the system options you need go into rc.conf (almost uniformly). Everything is documented and you can basically operate the system out of the FreeBSD Handbook documentation (it's not at all comparable to the "how to use a window or a menu" level intro stuff the linux provides). You can't do that when everything is furiously churning every release. everything just works, everything is just documented, it's an experience when you're coming from linux.

and that forum post from 2007 on how to tweak a service script is probably still valid, because BSD hasn't had 3 different init systems over that timespan etc.

just like "engineering is knowing how to build a bridge that barely doesn't fall over", engineering here is knowing what not to churn, and fitting your own work and functionality extensions into the existing patterns etc. like it doesn't have to be even "don't make a bigger change than you have to", you just have to present a stable userland and stable kernel interface and stable init/services interface. the fact that linux doesn't present a stable kernel interface is actually fairly sketchy/poor engineering, it doesn't have to be that way, a large subset of kernel interfaces probably should be stable.

I don't believe this will result in conservatism around upgrading nor that it would be good if it did. The vast, vast majority of security vulnerabilities are due to regular bugs and once the details are known, they can be exploited. Not patching leaves you open to these exploits.

Supply chain attacks are a real threat too, but being conservative about upgrading to improve your security posture is like saying "I'm worried someone with the key to my house might rob me, so I'm going to remove the lock from my door."

That's a double-edged sword. What happens when you need to upgrade in order to get vulnerability fixes?

The attackers will leverage any culture that helps them accomplish their goals.

If being rude and pushy doesn’t work, the next round will be kind and helpful. Don’t read too much into the cultural techniques used, because the cultural techniques will mirror the culture at the time.

The Jia Tan character was never rude. If you make rudeness the thing that throws a red flag, then ‘nice’ fake accounts will bubble up to do the pressuring.

I want to caution against taking a good thing too far.

There's a certain kind of talented person who is all too conscious of their abilities and is arrogant, irascible, and demanding as a result. Linus Torvalds, Steve Jobs, Casey Muratori come to mind. Much as we might want these characters to be kinder, their irascibility is inseparable from their more admirable qualities.

Sometimes good things, even the best things, are made by difficult people, and we would lose a lot by making a community that alienates them.

People have been bullied out of 'nice' communities. See the 'Actix' debacle in Rust.

I don't want to excuse rudeness or a sense of entitlement. But I think we can still understand where it comes from. A lot of these people probably work on crappy codebases where "let's just add a random dependency without any vetting" was the norm, they might have to deal with production issues etc. There's probably a systemic issue behind it, that our industry relies too much on unpaid labour and is usually not willing to contribute back.[0]

[0] Funnily enough, just a week or two ago, I fixed an issue in an OS project that we introduced at work. It was an easy frontend fix even for someone like me who doesn't do frontend and barely knows how to spell Vue. And more importantly, in the issue description somebody already wrote exactly what causes the bug and what would need to change - the only thing left was finding the place where to make the (one-line) change. Somehow that issue had been open for 2 years but nobody of the several people who complained (nor the maintainer) had bothered to fix it. After I made a PR, it was merged within a day.

Just don't do anything crazy. There are legitimately crazy people asking for crazy things, not necessarily backdoors.

Being rude is... unimportant. A lot of people think being passive aggressive is being polite when it's actually being rude + deceitful. There's nothing wrong with being direct, which some mistake for rude. I find it refreshing.

The issue here is the attackers will quickly move away from an individual attacking you to the group attacking you. The person writing the infected code will never be a jerk to you at all. You'll just suddenly see a huge portion of your mailing list move against you ever so slightly.

We've complained about bots in social media for a long time, but how many people in open source discussions are shady manipulative entities?

But in this case he was getting hit by both someone willing to help and then multiple people complaining that things were taking too long. And when you yourself feel like things are taking too long then you’re probably more susceptible to all this.

That sounds nice.

I did an engineering/program manager role for 8 years and people pretty much always did what I asked if I showed up at their desk or bothered their boss.

"Squeaky wheel gets the grease?"

But I too like to think that I prioritize my children on merit rather than fuss level. For some reason they continue to cry despite me claiming I don't react to it.

True, but a determined adversary like JiaTan/Jugar has an ace up their sleeve: they are good enough, and patient enough, to be able to fork the base project, spend a year or two making it better than the original (releasing the head of steam built up from legitimate requests that the old, overworked maintainer never got too, building goodwill in the process) and then convincing the distros to pick up their fork instead of the older original. At which point it really is game over.

It makes Rich Hickey’s „Open Source Is Not About You” [0] particularly poignant.

As a hobbyist developer/maintainer of open source projects, I strive to remember that this is my gift to the world, and it comes with no strings attached. If people have any expectations about the software, it’s for them to manage; if they depend on it somehow, it’s their responsibility to ensure timely resolution of issues. None of this translates to obligations on my part, unless I explicitly make promises.

I empathize with Lasse having been slowed down by mental issues. I have, too. And we need to take good care of ourselves, and proactively prevent the burden of maintainership from exacerbating those issues.

[0]: https://gist.github.com/g1eny0ung/9e7d4d0f72547a8d156452e76f...

Look how brilliantly they selected their target project:

(1) xz and the lib are widely used in the wild including linux kernel, systemd, openSSH; (2) single maintainer, low rate of maintenance; (3) the original maintainer has other problems in his life distracting them from paying closer attention to the project.

I am wondering how many other OSS projects look similar and can be targeted in similar ways?

The Jigar Kumar nudges are so incredibly rude. I would have banned the account, but perhaps they contributed something positive as well that isn't mentioned.

I wonder if it would be possible to crowdsource FOSS mailing list moderation.

I feel for Lasse.

It's time for more of the big vendors who use these projects in their offerings to step up and give people running these small projects more resources and structure. $20k to have maintainers for each project actually meet twice a year at a conference is chump change for the biggest vendors, especially when compared against the cost of the audits they'll now be doing on everything Jia Tan and Co. touched.

I’ve given semi-popular projects that I no longer had the bandwidth to maintain to random people who bothered to email, no pressuring needed. While those projects are probably four to five magnitudes less important than xz, still thousands of people would be affected if the random dude who emailed was malicious. What should I have done? Let the projects languish? Guess I’ll still take the chance in the future.

I'm reminded of the short story "The Strange Case of Mr Pelham", in which a man is stalked and eventually replaced by a doppelganger.

https://en.wikipedia.org/wiki/The_Strange_Case_of_Mr_Pelham

In "Ghost in the Wires" Kevin Mitnik details one of the ways he obtained information was via a law enforcement receptionist* who he managed to trick into believing he was law enforcement over the phone. He obtained information this way multiple times over multiple years, and fostered a phone based friendship with this woman. He seemed to have no qualms in doing this.

He was also turned on by multiple people who he considered close friends. In the book it did not seem that he had considered that it might not be a "them" problem.

*my details may be off here, I read it some time ago

It's bizzare enough as it is to start asking questions to confirm that "mental issue" had natural cause.

Thank the lord Lasse wasn’t maintaining the nuclear codes.

In another thought, I hope the nukes aren’t on a rolling Debian distro.

Not just for hardware support: https://github.com/serde-rs/serde/issues/2538

>It's either in source form and reproducable or it's not there.

Wanna know how I know you haven't read into the discussion much?

There are a whole lot of binary test cases in software. Especially when you're dealing with things like file formats and test cases that should specifically fail on bad data of particular types.

> It's either in source form and reproducable or it's not there.

Forbidding binary data in source repositories is neither practical nor solving the issue.

Binary code can be hidden steganograhically in e. g. image files or text files.

Are you running linux-libre?

You are implying that governments are more technically competent and more trustworthy than open source communities..

Avoid compromise with one simple trick: surrender to the attackers

> gov consortium

Personally I wouldnt trust a govt to not backdoor everything.

Where are the email addresses visible? I've also seen this a few times, but never the actual addresses.

I don't even think it's necessarily moderation so much as maintaining these high-traffic projects is now akin to a full-time job minus the pay.

At a certain point, companies need to step up and provide funding or engineering work or they should just keep expecting to get owned.

lcamtuf's two posts argue that this may simply not be an open-source maintainer's job to defend against. ("The maintainers of libcolorpicker.so can’t be the only thing that stands between your critical infrastructure and Russian or Chinese intelligence services. Spies are stopped by spies.")

That doesn't mean we shouldn't try to help burnt out committers, but the problem seems very hard. As lcamtuf also says, many things don't need maintenance, and just paying people doesn't address what happens when they just don't want to do it anymore. In an alternate universe with different leadership, an organization like the FSF might use donated funds to pay a maintenance staff and an open-source maintainer might be able to lean on them. Of course, that still doesn't address the problem of Jia Tan getting a job with this organization.

https://lcamtuf.substack.com/p/technologist-vs-spy-the-xz-ba... https://lcamtuf.substack.com/p/oss-backdoors-the-allure-of-t...

As a burnt out creator of open source projects with thousands of GitHub stars who's received numerous questionable maintainership requests: either very carefully vet people or let it stagnate. I chose the latter and just waited until others forked it, in large part because I didn't want to be responsible for someone hijacking the project to spread malware.

If I had ever received a request from a well-known figure with a longstanding reputation, who's appeared in-person at talks and has verifiable employment, I might've been more receptive. But all the requests I got were from random internet identities that easily could've been fabricated, and in any case had no previous reputation. "Jia Tan" and their other sockpuppets very likely are not their real identities.

This is not Lasse Collin’s responsibility. What is a burnt out committer supposed to do? Absolutely nothing would be fine. Doing exactly what Lasse Collin did and turn over partial control of the project to an apparently helpful contributor with apparent community support is also perfectly reasonable.

It's not a perfect solution, but I think Big Companies could have a role to play here. Some kind of tenure/patronage system, where for every $1B a Big Company makes in profit, they employ one critical open source maintainer with a decent salary (like $200k or something). The job would only have two requirements: 1) don't make everyone so mad that everyone on the Internet asks for you to be fired, and 2) name a suitable replacement when you're ready to move on. The replacement would become an employee at Big Company, which means Big Company would need to do whatever vetting they normally do (background checks, real address to send paychecks and taxes, etc).

In this scenario, Jia Tan would not be a suitable replacement, since they don't friggin' exist.

Yes, there's problems with this approach. When money gets involved, incentives can get distorted. It limits the pool of acceptable maintainers to those employable by a Big Company. But I think these are solvable problems, especially if there's a strong culture of maintainer independence. It provides a real improvement over the current situation of putting so much pressure on individuals doing good work for no benefit.

This is the problem - we have record numbers of computer programmers, having increased exponentially from the 90s, however very few of them want to work just for fun on some library buried deep in the system, most are just out for a quick buck.

Look at the number of people who make meaningful commits to ffmpeg vs the number of people that wrap it in some UI and somehow persuade thousands of people that their solution is amazing, and ffmpeg is a sexy project.

Until we can patch humans, social engineering will always work. Burnt-out comitter or not. Just be vigilant.

The same as what the ant colonies do when one of their own gets infected by the Ophiocordyceps unilateralis fungus.

That's a great question and instinctively I'd say better to halt development than cave to pressure.

Check the GitHub profile of anybody that commits. Is there a photo of the person? Can you see a commit history and repos that help validate who they seem to be.

In this instance, noticing the people emailing to pressure you have fake looking names that start with adjacent letters and the same domain name.

Be more paranoid.

get the project taken over by a foundation eg the Apache Foundation.

tell people to fork your project and fuck off

Force pushes tend to be noticed easily. All it takes is for one external developer to try to pull to see the failure. And it's actually hard to do because you need to comb through the tree to update all the tags that point to the old commits. On top of that it obviously breaks any external references to the commit IDs (e.g. in distro or build configurations), all the way up to cryptographic signatures that might have been made on releases.

I think it's a pretty reasonable assumption that this didn't happen, though it would be nice to see a testimony to that effect from someone trustworthy (e.g. "I restored a xz checkout from a backup taken before 5.6.0 and all the commit IDs match").

The way the hack works is incredibly sophisticated and has specifically sought out how to get past all normal checks. If messing with some commits would be possible this entire rube goldberg hack would not have been set up.

There are trusted copies of historic releases from third-party sources (at least Linux distributions, but there's probably other sources as well), it's pretty easy to check whether the tags in the git repository match those. (This can be done as the tarballs are a superset of the files in the git repository, the other way around doesn't work).

What about just using 'web of trust', for example with GPG? If the user's key is signed by people that met up with the actual person, it would be much harder to make fake identities.

I think it is unauthenticated from the point of view of SSH’s own authentication. The backdoor has its own credential, but the RCE is accessible if you don’t have an account on the system.

"unauthenticated remote code execution" is a fairly standard term for this kind of access.

(https://www.google.com/search?q=%22unauthenticated+remote+co...)

Yes the whole "let's take a mystery meat tarball from a repo that isn't the project repo" seems suspect.

Github+ even has a scheme for signing artifacts such that you have some level of trust they came from inside their Actions system, derived from some git commit. This would allow the benefits of a modular build for a large product like a distro, while preserving a chain of trust in its component parts.

+Not advocating a dependency on Github per se -- the same sort of artifact attestation scheme could be implemented elsewhere.

I think that trust needs to be 'pushed deeper' than that so to speak. While this would be an improvement, what happens if there is a malicious actor at Github? This may be unlikely, but would be even harder to detect since so much of the pipeline would be proprietary.

Ideally, we would have a mechanism to verify that a given build _matches_ the source for a release. Then it wouldn't matter where it was built, we would be able to independently verify nothing funky happened.

Wasn’t the payload in a blob in the tests, which is in the source repo? If you were to clone the repo then build from source, you’d have the backdoor, right? Surely distros aren’t using binaries sent by maintainers

Open source fundamentally does not work that way. There are many important open source contributors who work pseudonymously.

Google's Know, Prevent, Fix blog post floated the idea of stronger identity for open source in https://security.googleblog.com/2021/02/know-prevent-fix-fra... and there was very significant pushback. We learned a lot from that.

The fundamental problem with stronger identity is that spy agencies can create very convincing ones. How are distros going to detect those?

What would prevent a known person from accepting a govt payout to sabotage their project, or to merge a plausible-looking patch? Relying on identity just promotes a type of culture of reputation over code review.

Nope, identities won’t solve it, you can have people coerced, blackmailed, threatened, or simply just a “front” while there’s a whole team of spies in the background. The process should be about what’s being pushed and changed in the code, but I would be lying to say I have a concrete concept how it is possible.

If this was done by a state actor then this policy wouldn't help at all. States have no shortage of identities to fake.

How would that even work? Are distros expected to code their own alternative versions of open source libraries where they can't get the maintainers to send their IDs? Or what stops from forged IDs being used?

This will never be accepted by the community.

I don't think these questions are that interesting. SSHD shared an address space with xz-utils because xz utils provides a shared library, and that's how dynamic linking works. sshd uses libsystemd on platforms with systemd because systemd is the tool that manages daemons and services like sshd, and libsystemd is the bespoke way for daemons to talk to it (and more importantly, it is already there in the distro - so you're not "adding a million line dependency" so much as linking against a system library from the OS developers that you need).

Linking against libsystemd on Debian is about as suspicious as linking against libsystem on MacOS. It's a userspace library that you can hypothetically avoid, but you shouldn't.

As for why systemd links against xz, I don't know, and it's a bit surprising that an init system needs compression utils but not particularly surprising given the kitchen sink architecture of systemd.

It's certainly a reasonable question to ask in this specific case. In hindsight Debian and Red Hat both bet badly when patching OpenSSH in a way that introduced the possibility of this specific supply chain attack.

> If so, we should be proactively removing and locking down other dependencies, because it will likely be an effective and simple mitigation.

I think this has always been important, and it remains so, but incidents like this really drive the point home. For any piece of software that has such a huge attack surface as ssh does, the stakes are even higher, and so the idea of introducing extra dependencies here should be approached with extreme caution indeed.

Another point relevant on the timeline is when downstream starts using binaries instead of source.

I think people are flying past that important piece of the hack. Without that this would not have been possible. If there is a trusted source in the middle building the binaries instead of the single maintainer and the hacker this attack becomes extremely hard to slip by people.

It does, but has there ever been a movie that successfully portrayed a compelling drama that takes place entirely on a computer monitor? It's hard to even imagine. It's why I think the novel is still relevant in our age because all the great stories that unfold on a screen can't be acted out on a sound stage.

It’s not compelling enough unless we find out who was behind it, which is probably unlikely.

I’m of the opinion that there are backdoors in most of our software and a lot of our hardware. xz just happened to be caught because it was hogging resources.

Thanks for this comment. I've added that LKML patch set to the timeline.

I don't think the attacker saw the systemd change at all personally.

The way the exploit was setup, they could have pivoted to targeting basically any application server because there's so many interdependencies. python, php and ruby could be targeted because liblzma is loaded via libxml2 as an example.

Gaining trust for linux kernel commits would have just let them continue maximizing profit on their time investment.

>particularly if the attacker intended to target embedded systems (such as military transport vehicles [6])

Said vehicles aren't networked on the public internet and from experience in this particular sector, probably haven't been nor will be updated for decades. "Don't break what isn't broken" applies as well as "military doesn't have a budget to pay defense contractors $1 million to run apt-get update per vehicle".

> target embedded systems (such as military transport vehicles

You are giving a lot of credit to that, I have seen military ones with ancient software, even the “new” updated ones are still on ubuntu 18.04 because of some drivers/sdks compatibility, but it isn’t a major issue since most of the times they are not connected to the public internet.

I don't understand how this could have worked.

If you compile and build your own image, would that be able to trigger the backdoor?

You can of course change an existing image to something that triggers the backdoor, but with that level of access you won't really need a backdoor, do you?

I wouldn't be surprised if there are others but this specific one seems special. It was caught because on connection it does an extra decryption operation and I'd assume there is no way around this extra work. They'd have to re-architect this to not require that decryption.

I'm not a security expert though.

we were discussing this on the IRC. Imagine spinning up a thread, then running a bsd style pledge(2) on it to call liblzma. Kinda janky but it would work. Another option would be to just go out and call the xz util and not rely on a library to do so. That process can be locked down with pledge to only have stdin/stdout. That's all you need.

So, like UNIX does have this plumbing, its just that reaching for libraries and tight integration has been the pursuit of Lennart Poopering and his clan for years.

I would be very surprised if there wasn't a Federal-level investigation being planned (or already ongoing) into this. Of course it's always a possibility that the attack was, in fact, sponsored by these same agencies.

Could've happened just as easily if not more easily with proprietary software.

Can you explain what aspects of the open source philosophy were exploited, and what possible mitigations might be?

I get that there is a reason not to trust those Patches, but I would guess they don't contain anything malicious. This early part of the attack seems to only focus on installing Jia Tan as the maintainer, and they probably didn't want anything there that could tip Lasse Collin off that this "Jia" might be up to something.

In https://archive.softwareheritage.org/browse/revision/e446ab7... one can open the patches and then click the "Changes" sub-tab. Stuff like this looks like a perf improvement but who knows if a tricky bug is introduced that was aimed to be exploited https://archive.softwareheritage.org/browse/revision/e446ab7... There are more patches to be vetted unless one would give up and say that 5.2 should be used as last "known-good".

> And all because one owner of a library had some mental health issues

Wrong takeaway.

Impossible here because the exploit was carefully engineered to be unreplayable and NOBUS (nobody but us) so it couldn’t go viral. Even if you intercepted a complete tcp byte trace of the attack there was nothing you could do with that to attack other systems.

There's a single dot in a line between #include <sys/prcntl.h> and void my_sandbox(void). It is easy to miss, but makes the compile to fail, thus resulting in HAVE_LINUX_LANDLOCK to be never enabled.

> How do you get "Jigar Kumar"'s email address?

Hit reply

I think the period on the line above void my_sandbox.

> Timeline reads like a "pig butchering"

To me this is the polar opposite of pig butchering. This was a targeted and unromantic attack, unrelated to investing or cryptocurrency, the original maintainer was not "fattened like a hog" in any way, if anything he was bullied and abused into submission.

I'm amazed more people aren't talking about the "off list" part or asking Colin if he's willing to provide those emails/conversations.

Not OP, but the fact that they only appear in this context makes me believe that their are specifically set up for this task. No regular person has that good opsec just to push patches to a random library.

If the account was compromised only recently the real co-maintainer would have done everything to warn the maintainer. If you're maintaining some core piece of infrastructure and your account gets compromised it would be trivial to let at least someone know you're not the one pushing these commits.

Completely benign documentation change to fix a typo: https://github.com/MicrosoftDocs/cpp-docs/pull/4716

I have no idea what that IP address is supposed to be about...

So all binaries built with a Microsoft compiler must be considered compromised?

Well, test files shouldn't be affecting the actual production binary.

But in practice that's not something that can be enforced for arbitrary projects without those projects having set something up specifically.

For example, the project could track the effect on binary size of the production binary after every PR. But then it still requires a human (or I guess an AI bot?) to notice that the increase would be unexpected.

Also that when dynamically linking A against B, A apparently gets free reign to overwrite B.

It sort of makes sense, since at the end of the day it could just be statically linked or implement B's behaviour itself and do whatever it wants, but it's not really what you expect is it.

He put this comment because he knows that FOSS enthusiasts and especially Debian always prefer source over binary. This is not only true for program code, but also includes docs, images etc.

The correct way to do that would be a source that generates a test file and then a script which reproducibly produces the desired corruption.

Chronic mental health issues are extremely common.

Not being paid for your work while others profit from it surely doesn’t help with one’s mental state.

This event should be a wake-up call to businesses everywhere: It's not just a small number of "core" FOSS projects that need their support (funding and assistance!). Before this event who was thinking about a compression library when considering the security of their FOSS dependencies?

The scope of "what FOSS needs to be supported and well-funded" just increased by an order of magnitude.

Never understood why our industry seems unique in our willingness to do unpaid work for giant corps. Your compression library isn't saving the world, it's making it easier for amazon to save a few bucks.

You have the right to be paid for your time. It's valuable.

I enjoy coding too... but the only free coding I do is for myself.

Use a proper license, charge for your time and stop killing yourself doing unpaid hobby projects that cause nothing but stress.

I think a problem is that there doesn't seem any way to automatically check this. If we assume that anything that is used during build time can be malicious then figuring out those dependencies is already hard enough. Mapping that to organizational stability is one step further.

What it reminds me of, is my reoccuring thought, not just about open source but including this aspect, that we've built up a society based on software, that we could literally only afford because we've done it unsustainably. The economy could not bear the costs of producing all this software in an actual reliable sustainable way. So... now what.

Nothing will really change, sadly. Remember log4j? There were also a lot of talking why people working on FOSS should be paid. And after one month almost no one remembered these voices exept for small minority of people.

That's just the nature of free software- you have to either trust the maintainer or do it yourself. There is not really a way around it.

- Corporate maintainers are great until they enshittify things in the pursuit of profit (see Oracle)

- Nonprofits are probably the best but can go the same route as corps (see Mozilla)

- Hobbyists are great until they burnout (see xz)

What's gonna happen now ? a team of foss sec chaos monkey trying to run checks on a core set of libs ?

Sorta but flawed relevant xkcd: https://xkcd.com/2347/

I don't see how this is any strongly different than some unappreciated skill worker in a corporation. Its interesting the double standard we have for FOSS. Meanwhile in the commercial world, supply chain attacks are commonplace and barely solicit headlines.

Yes, FOSS needs to be able to address these kinds of attacks, but the world runs on the efforts of the low-level few, generally. The percent of people who work to build and maintain core infrastructure has always been small in any economic system. The world is held up by the unsung labor of the anonymous working class. Think of all the people working right now to make sure you have clean water, electricity, sanitation, etc. Its a tiny fraction of the people in your city.

Conversely, why aren't all these corporations who depend on this contributing themselves? Or reaching out? There's a real parasitic aspect here that gets swept under the rug too.

I'd even argue this isn't really a hobby for many, especially for higher profile projects. For many its done for social capital reasons to build up one's reputation which has all sorts of benefits, including corporate advancement, creating connections for startups, etc. Its career adjacent. And that's ignoring all the companies that contribute to FOSS explicitly with on-the-clock staff.

So there are motivators more than just "I'm bored and need a hobby." Its a little dismissive to call FOSS development just a hobby. Is what Linus does a hobby? I don't think most people would think so. Things like this have important social and economic motivators. The hypothetical guy in the comic isn't some weirdo doing something irrationally, but has rational motivators.

I'd also argue that its pretty harmful to FOSS adoption if the community takes on a "well, its a hobby don't expect quality, security, or professionalism." This is a great way to chase people away from FOSS. We can't just say "Oh FOSS is better than much closed software" when things are good, then immaturely reply "its just a dumb hobby, you're dumb for trusting me," when things go south. I think its pretty obvious there's a lot of defensiveness right now and people being protective over their social capital and projects, but I think this path is just the wrong way to go.

Comms, PR, and image management in FOSS is usually bad (see Linus's rage, high profile flame wars, dramatic forkings, ideological battles, etc), so optics here aren't great, because optics is something FOSS struggles with. The community is at best, herding cats, with all manner of big personalities and egos, and its usually a bit of a controlled car crash on the best of days.

Also in security and 100% tired of the code slingers who get annoyed by security reviews

Here on HN it’s been derided as “company just checking a box for compliance but adds no functionality. It slows us down when we want to disrupt!” - developer of yet another todo list or photo editor app…

Buffer overflows and the like are one thing. Notions from this blog that certain normal files won’t be well reviewed is a bad smell in software. Innocuous files should be just as rigorously reviewed as it’s all part of the state of the machine

“This is how it’s always worked” is terrible justification

Startup script kiddies git pulling the internet, and single maintainers open source projects aren’t cutting it; if it’s that important to the whole those in charge of the whole need to make sure it’s properly vetted.

I’m an EE first; this really just makes me want to see more software flashed into hardware.

Ah, the old "undercover cops can't lie about not being a cop, just ask them" technique.

What makes you think they are an actual person?

Completely conclusive proof? No, but it seems unlikely that there ever would be conclusive proof of such.

They don't seem to exist outside of this incident and things related to this.

There were multiple people who also don't seem to exist outside of their posts to the xz mailing list applying pressure for the original maintainer to bring Jia on board. This occurred around the time that Jia was first making contact with the project, not only recently.

Apparently the IP addresses that were logged for Jia is a VPN based in Singapore

They have vanished.

Honestly, there's very little evidence that they weren't always intending this.

> It seems a lot of people are assuming Jia Tian was compromised all along. I haven't seen a reason to believe this was a long play, but rather that they were compromised along the way

So I assume Jia has spoken out since? How do you go this long without realising someone else is making plays as you?

I took a look at Jia Tan's early behavior, and I found it to be consistent with being "compromised" from the beginning. They had months of contributions on private repos before forking a test library and making superficial changes to it, and then diving headlong into archival libraries. It all looks set up and I see no evidence of an actual person at any point.

I also think it is more difficult to get away with bribing/beating an existing contributing than you suggest; esp since failure means likely exposure.

Regarding point 1: The timeline in the linked article describes some communications as "pressure emails." I've heard the theory, but haven't seen solid evidence, that the pressure emails weren't just regular impatience from outside devs, but actually part of the attack. To convince the primary maintainer into granting access to another party.

Yeah whatever. Closed source software is much easier to subvert, just have your agents join the company and they can push whatever they want without any external (or even internal) review.