> Open source is a supply chain risk Given the potential impact of OMIGOD, the million-dollar question is “How does an open source component with only 20 contributors on GitHub affect the security of large portions of Azure?” The ease of exploitation and the simplicity of the vulnerabilities makes us wonder if the OMI project is mature enough to be used so widely.
> Yet this scenario is more common than you might think and certainly not unique to Microsoft. One of the benefits of open source is that it’s easy for developers to grab code from different projects and mix it with other open source and proprietary software. As a result, bad open source code can wind up in an enormous range of products and services – inadvertently becoming a “single point of failure.”
The fear mongering about open source in the article seems misplaced.
Any software product could lead to the same thing, and OMI was used as a software product in this instance. I don't see how it being open source changes anything and the article doesn't address this, only goes on a tangent about how "open source code can end up in any software project".
It also leaded with a bit about supply chain attacks, in a way that seemed to imply, without explicitly stating, an intentional cause to this exploit existing.
> Because customers don’t know what franken-code is running in the background of the services they use, they remain at risk and unaware.
Wouldn't this be worse with closed source software thou?
Ironically, the OMI 'open source' package is Microsoft's.
The only recent commit was for "Enhanced security" a month ago:
Indeed! And most of the 20 contributors work for Microsoft. Talk about FUD.
I read all of this: https://github.com/microsoft/omi/blob/master/README.md
still don't understand what it does
It’s part of dumpster fire that is the DMTF ecosystem. Attempting to understand it if you haven’t been exposed to Windows enterprise bullshit is futile. It’s layers upon layers of meta XML to do the most mundane things in the most overcomplicated way without actually succeeding at making any usable standard.
Or, in other words: if you reaction to SOAP was “damn, this is amazing, we need more of this but with even more indirection”, you’ll feel right at home.
Ars Technica had an article on three vulnerability that was decently clear: https://arstechnica.com/information-technology/2021/09/secur...
(Sounds like it's a Linux port of some Windows management feature.)
This is one reason I stay the hell away from Windows focused software on Linux. There’s a complete impedance mismatch. Also MSFT’s GitHub open source fairy dust project stewardship is a shit show from experience.
yes, checking the secret before setting the uid/gid does seem like "Enhanced security" : https://github.com/microsoft/omi/commit/4ce2cf1cb0aa656b8eb9...
Exactly. It's not like Microsoft picked up a random unaudited open source project and shoved it into all Azure VMs. OMI is maintained by Microsoft and the fact that it is open source is a good thing security wise. I'd wager being able to look at the source code greatly helped the researchers in their diagnosis as well.
At some level, agreed that it probably helped security wise for researchers. That the fix was committed in public and then vulnerability remained for a month and well still currently for customers is where one could make a case that the open source nature did as much damage as good.
> That the fix was committed in public and then vulnerability remained for a month
That's an indication that Microsoft is bad at handling security issues. Which isn't news and has very little to do with open source. Something very similar happened with Exchange, it led to plenty of people being compromised, and Exchange is not open source.
When you distribute patches for a vulnerability it basically makes no difference whether the source for those patches is available - people are quickly going to figure out what the vuln is. This is a patch rollout problem and nothing else.
$ apt update
Oh, wait, fix not available in the Debian/Ubuntu official repos, only M$ trusted ones.
Indeed, it is quite ridiculous to lay the blame for this on an open source project. They did not force Microsoft to use their software and deploy it to all their customers. It is entirely Microsoft's decision to accept software that waived all warranty, liability, and guarantee of fitness for purpose and then deploy it to all of their paying customers without doing appropriate due diligence.
Any company in any other field that accepted dependencies without doing quality checks to verify that those dependencies met their requirements would rightly be called grossly incompetent. This is one step further than even that as the dependencies openly guaranteed nothing and they still did not verify if it was of acceptable quality before making vast amounts of things depend on it as a critical component. This is the absolute basics of the basics and Microsoft has completely failed at even that. This is indicative of deeply-rooted gross systemic incompetence in much the same way that failing FizzBuzz demonstrates a complete lack of programming ability. As such, their opinion on matters of security and their ability to deliver security should be completely disregarded until their organization is completely overhauled.
What? This is Microsoft's open source project - they didn't need to "do quality checks", they developed it!
Your comment makes no sense to me - you're saying we shouldn't blame open source but in the same breath say that anyone who uses open source without carefully reviewing and auditing every line of code is an idiot who can't even do FizzBuzz. I'm a developer and if I'm faced with the choice of auditing every line of code in a dependency or just writing what I need I'll just write what I need nine times out of ten. Honestly your position is closer to the article than you seem to realize.
Yes. The explanation given under that rather provocative headline does not mention how the OMI project being open source, as opposed to closed source, enhances its status as a supply chain risk. Microsoft could have just as easily not released that code - it still would have included the vulnerability.
Open source supply chain is a real thing, but not sure it's at all relevant in this case. In this case it appears to be an open source package that was simply open sourced by Microsoft as opposed to some broader community project.
That it's open source there could an interesting argument made that it's easier for one to hunt for vulnerabilities when they can review the code. In this case what was the advantage or point of it being open sourced when it is clearly a Microsoft project for Microsoft purposes. Maybe for collaboration internally? But the open source nature of it doesn't gain much.
The open source nature–in this particular case becomes more complicated when the fix is in plain sight for a month now with a seemingly easy ability to take advantage of. So in this case closed source fix that wasn't public how to exploit maybe would have been better indeed.
The open source vs. not is a bit more of a red herring relative to the overall situation. It's certainly not the central theme, but when you overanalyze I can see how some of the analysis maybe happened.
The fear mongering of open source is fabricated by the very people whom intentionally include vulnerable code in proprietary software running on every device. They don't want you to see it.
> As a result, bad open source code can wind up in an enormous range of products and services – inadvertently becoming a “single point of failure.”
What's the alternative? A black-box proprietary bug that becomes an NSO or NSA exploit?
The logic on this is so bad.
Linus's law: "given enough eyeballs, all bugs are shallow".
Exactly, they were doing fine until they made a point to shit on all open source software instead of limiting their focus to this one, Microsoft-sponsored outfit.
"Currently, this project is not accepting contributions or pull requests."
Last changed Aug 17, 2016.
There are also people who think open source is the holy grail of security because "given enough eyeballs, all bugs are shallow". And is used as an argument. So it cuts both ways.
"given enough eyeballs, all bugs are shallow" Is certainly true, but that doesn’t make OSS the holy grail.
In fact, I’ve never met anyone who thinks that. And “all bugs are shallow” doesn’t mean there are no bugs, it just means it’s easier to find bugs.
The thing about dependencies is most people don't actually read the source. But at least you have the option.
Yeah I thought the list of maintainers for bash and curl were pretty short (like one or two people each).
It's hilarious we're using the actions of microsoft as an example of opensource failing.
The apparent fix: https://github.com/microsoft/omi/commit/4ce2cf1cb0aa656b8eb9...
Looks like a standard, absolutely gigantic C/C++ project that re-invents absolutely everything including an event loop, http server, XML parser, a C++ tool to generate C code to create a “str hash” implementation, a custom lexer for WQL and a parser for a “mof” file format.
Still not clear on what it does. Seems safe.
> a standard, absolutely gigantic C/C++ project that re-invents absolutely everything including an event loop, http server, XML parser, a C++ tool to generate C code to create a “str hash” implementation, a custom lexer for WQL and a parser for a “mof” file format.
This is the part that gets missed when people talk about "dependency inflation" re: Rust. Absolutely, it is a problem, but most C codebases of a sufficient size and vintage are vendoring some absolutely insane hidden "dependencies" that, on average, probably get a lot less testing and attention than the average package on crates.io.
There are definitely risks with both approaches.
In C it is much more inconvenient to include external dependencies, so it's much more common for people to roll their own implementations rather than using a tried-and-tested one.
Or copy in a .C/.h file that gets forgotten about, and therefore gets no security updates.
CIM/WBEM goes all the way back to 1996. They essentially wanted a management infrastructure on all kinds of devices (including different architectures, so actually C made sense then), but that also notably included remote access. At the time, SOAP was still popular, so here we are with a rather silly transport protocol and all kinds of overhead reinventing things like SSH. However, the overall goal still makes sense, it was essentially a way of 'object'-ifying everything from logs to other metrics. This fit in with the overall mode of thinking in MS with DCOM and COM (and registry), and structured configuration/management. I'm sure it's paid massive dividends on Azure Linux infrastructure. For highly structured objects, SOAP and XML aren't a terrible fit, but I doubt many people would do the same thing again today.
Honestly, they just needed to rewrite it in a safer stack. However, that still may not have saved them from all these vulnerabilities, given the scope of what they're implementing as remote management protocols. The relative scrutiny, fuzzing and manpower just hasn't been there, especially when it's obfuscated by various layers.
Not to take away from the rest of what you said, but I don’t think SOAP was _still_ popular in 1996. I don’t think it had become popular yet. I don’t think I even heard of SOAP before 1999 or 2000. I’m not a trend setter or anything, but if it was popular, I probably would have at least heard of it.
Greenspun's law, "Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp." is really just a special case of a general principle. There's a lot of cases where if you use the wrong tool and/or don't know the right libraries, or don't or can't use them, or just won't, that you'll reinvent something that already exists, badly.
But does it receive email?
Looks like it’s mostly C, and not great C at that… In my humble opinion, this is a scary looking project.
Specifically- it appears to be mostly C99, lots of macros, lots of ugly buffer manipulation. But it's tooled for PREFast so you know it's secure! (https://github.com/microsoft/omi/blob/4ce2cf1cb0aa656b8eb934...) (j/k... I haven't seen that in ages). Guessing a port of MS's internal WMI codebase?
This codebase just smells of rot all over.
That is why there is "Modern C", "Modern C++" as talked at conferences, blog posts and endless over here, and then real life code that we actually find out in production.
Sure, yet this is the outcome from companies deeply involved in ISO, let alone those where regular folks don't even care HN exists.
It can also be RIIC#, RIID, RIIGo, RIIAda, RIIJava,... take your pick.
In typical Microsoft fashion, regardless of how much victories .NET and now Rust dev units achieve, the OS ones keep pushing for C and C++ no matter what.
That is also why we now have WinRT, WinUI, Sphere OS, Azure RTOS, regardless of what Microsoft Security advises.
The "fix" doesn't include any tests that would indicate that it actually works. Doesn't seem like a very healthy project.
The explanatory commit message of "Enhanced security" should be enough to document all you need to know!
You'd think a mention of the CVEs involved is mandatory for commits like these.
After the JEDI black eye and slow attrition of frustrated developers/CIOs, this feels like a huge blow. Cloud is all about trust. They broke trust by installing the agent without disclosures, and shattered it with this vulnerability. Their customers running Linux vms have a pretty simple migration path to other cloud/hosting vendors, and most people have learned that only having one approved vendor for cloud just doesn’t work. Q4 and 2022 could be real rough for MSFT at least within the scope of affected customers. Moreover the torture that is Teams has got to have consequences as millions have had to suffer, and Windows 11 is a dumpster waiting to catch fire. My Threadripper 1950X can’t run Win11, and Edge is so difficult to detangle it’s practically antitrust-worthy. Oh, and setting up hundreds of thousands of developers to unknowingly commit licensing violations via Copilot…
Whatever magic Nadella brought to revive the company is starting to wear off, at least from where I’m sitting.
> JEDI black eye
How much of that is due to the new administration not liking the previous one?
> slow attrition of frustrated developers/CIOs
> Moreover the torture that is Teams has got to have consequences as millions have had to suffer
Then why not use one of the 10 competitors for chat apps?
This can't be real, I thought. This is just a stub in a random commit somewhere.
But no. Same code is on master branch atm.
What is this, a joke?
"Never attribute to humor, that which is adequately explained by incompetence"
I'm going to give some benefit of the doubt and assume¹ that is some sort of stub for dev/test, replaced by one of a selection of proper symmetric encryption options in any production use. Amusing anyway.
¹ Anyone with more spare time than I want to look deeper to confirm or contradict?
It's used there: https://github.com/microsoft/omi/blob/e4d72481fa2f805148c9c8...
Also note that neither EncryptData nor DecryptData have any way of passing a key in their API. So it's very unlikely that these implementations could be conditionally compiled in place of a real implementation (but they're the only definition of these functions in the repository anyway).
It used to have actual Windows specific crypto code which has been removed in the linked commit.
I assume this has been ported from Windows and later never implemented the ripped out components. That said, I don't know the windows API so apart from confirming that they exist in Windows docs I can't assess how valid their usage was.
> neither EncryptData nor DecryptData have any way of passing a key
There could have been other smells involved, like the key being held in some more global scope instead of being passed in via the call stack.
Let's hope memcpy hasn't been #defined
At least it's fast. :D
And properly SAL annotated too.
An encryption scheme with truly memcpy-like performance characteristics.
No surprise to be honest.
What rather surprises me are the comments claiming MSFT is suddenly going to go bankrupt because there is a pre-install spyware.
Have you ever worked in a Fortune 500 ?
Do you know how hard it is in those company to get anything done that has not been budgeted and planned years in advance ?
Do you seriously believe Fortune 500 using MSFT are going to suddenly migrate ALL their Azure workload somewhere else like some kind of startups run by 3 devs ?
Yes Azure VMs have spywares builtin , but last time I checked all Europeans companies that are using American Cloud fall under the « Cloud Act » legislation which « legally » requires cloud vendor to hand over ALL the data the company is currently hosting.
I’ve worked in some of the largest insurances in Europe they LOVE Microsoft and Azure , even if there is this spyware issue , they will pretend it doesn’t exist and do business as usual.
Im pretty sure everyone will forget about this news in 1 month or so.
It blows my mind how easy it is to get root with this - "Simply remove the auth header and you are root."
That the kind of thing you get with homegrown web servers and corporate deadlines. I see how a QA person wouldn’t test that case, it’s almost a base assumption that the header would be mandatory.
Wiz is definitely trying to drum up some excitement around their new business with all these recent disclosures. Related but slightly off topic, we pay an industry analyst for high level market research and he's been screaming from the building tops about Wiz.io for the last 6 months. I'm honestly tired of hearing about it, he must get a cut of the sale. I wonder how long Wiz will keep this up; until they get sold presumably, then they'll go the way of the other big cyber firms and tone down the big hacks.
If you want to see what all the hype is about - I can try arrange a demo meeting for you ;) email@example.com
Peeking in from the sidelines, as someone who probably will want to arrange a demo too, here's a thing I've been scratching my head with. Your online pitch contains a logical puzzle I haven't been able to decipher.
Patch management and vulnerability checks, right. (VM) OS configuration checks, sure, why not. "No agents or sidecars to deploy" ... hang on, how does that combine with the other two? In order to check for installed/missing patches for on-system software you have to have some kind of access to the underlying system.
Surely you are not snapshotting root volumes for your analysis needs?
Very slim chance it’ll get sold. It’s going for an IPO to play alongside other cyber security companies like Sentinel One. The founders are already rich enough from previous ventures.
While I think this article is unnecessarily critical of the Azure OMI agent, this is a very "What the heck, Microsoft!?" moment for me. Of all the pieces of Azure infrastructure, the OMI agent is absolutely something I expect to be well-tested and secure.
I recognize that bugs happen, but allowing a remote client to execute commands as root by simply removing the authorization header should have been caught by automated testing.
This announcement seems badly timed given the OMIGOD vulnerability:
"Microsoft announces passwordless future – available across Microsoft Edge and Microsoft 365 apps": https://blogs.windows.com/windowsexperience/2021/09/15/micro...
No, see, it's totally consistent. Instead of just hackers having passwordless access, now everybody can!
/s, if it wasn't obvious...
GCE's counterpart doesn't seem to have a public endpoint and its functionality seems make sense: https://github.com/GoogleCloudPlatform/guest-agent
I have to say the problem is not oss, not agents, but Microsoft.
https://github.com/Azure/WALinuxAgent - I think this is the equivalence of GCP guest-agent, serving similar functionalities, and is pre-installed on all official images, otherwise basic things like authentication and image baking will break.
By setting the provisionVMAgent property to false when creating a virtual machine, WALinuxAgent should run with all extensions disabled, and I think that's as minimal as a Linux VM can go on Azure.
This property, however, can't be set via https://github.com/ansible-collections/azure, which is of course another lovely OSS project by Microsoft. I didn't bother to send a PR.
The OMI agent seems to be a different beast that is way more obnoxious. The closest thing on GCP is probably the collectd agent and the fluentd agent installed for Stackdriver Monitoring and Stackdriver Logging? Plus whatever OS config to enable unattended upgrades.
I just learnt from this HN thread about the SSM agent on AWS. That one does seem equally obnoxious as the OMI agent.
EDIT: Looks like collectd and fluentd have been replaced by https://github.com/GoogleCloudPlatform/ops-agent
And it is written in Go. I think it is generally a safer choice.
https://github.com/GoogleCloudPlatform/compute-image-package... - it used to be just one project with Python, C++, and shell scripts. The Python bits got rewritten into Go, while the others got split into their own repositories.
Wait, is Azure not patching this on customer systems? They're leaving it up to customers to patch a hole they themselves introduced?
That immediately makes me wonder about AWS SSM, which I spotted preinstalled on a Ubuntu 18.04 AMI the other day. I think it can’t be accessed outside of the VPC but it’s worth further research. Agents are scary.
>...an open source component with only 20 contributors
How's that not a lot of people? I don't get the "only" at all.
Is this meant to be a joke?
I was surprised by this as well. 20 people is a lot for a project, especially an open source project.
I would have liked to see what the author thinks is a sufficient number of contributors.
I’m pretty cynical because I’ve frequently seen a lot of pot calling the kettle black accusations when the authors projects are shown to only have 1 or 2 contributors.
This is the second Azure disclosure from the guys at Wiz, following the one with Cosmos DB few weeks ago. Now, it’s much more interesting when you take into account that Wiz was founded by the guys who sold a cyber security company, Adallom, to Microsoft and then served in senior roles around Microsoft cloud security post acquisition. Assaf Rappaport, Wiz CEO, served as fhe GM of the Cloud Security Group at Microsoft for five years .
I wonder what the people at Microsoft are thinking of this situation.
I'm sure its awkward but they appear to be an equal opportunity reporter - https://www.wiz.io/blog/black-hat-2021-aws-cross-account-vul...
Their product is pretty interesting. I did a demo a while back and their approach of building a large graph and doing a good amount of reachability (network and entitelement) leads to some useful signal. It's not entirely novel, JupiterOne has been doing something in the same vein for years now but it seems to work well. Still feels a bit rough around the edges but i thought it was interesting and could work well in situations where simple checkbox-style config analysis falls short.
I was able to leverage JupiterOne to identify the misconfiguration mentioned in the Wiz article across all my AWS accounts in a single query. Pretty nifty.
They shared that query and a bunch more in this blog: https://try.jupiterone.com/my-bucket-my-data-or-is-it
Interesting. Would be curious to know what parts they worked at. GM seems high enough to know about "areas to look at".
Still doesn't excuse having such bugs in the first place though.
What's the implication? That faults were included or neglected to be sussed out by their next company?
Absolutely not. But it’s still kinda weird. Like Facebooks CISO leaving their post to found a company which promotes itself with Facebook CVEs. To quote Matt Levine, “I don’t know”.
> Now, it’s much more interesting when you take into account that Wiz was founded by the guys who sold a cyber security company
security people sell security company and create a security company. color me shocked.
Nice out of context quote.
I am using Azure on one project as per client's requirement.
One thing I can say, Microsoft hasn't changed a bit. It's a Win98 experience in the cloud.
Now we are fighting Microsoft silently blocking entire ASN of Airtel in Nigeria, and:
1. First, obviously pretending having no idea what are we talking about, and bullshitting us
2. Then not acknowledging. "We checked it, everything is fine"
3. When faced with traceroute - "Maybe it's a third party network provider"
4. When faced with WHOIS record - "I don't know what really to do"
5. "When faced with "this is the last day we are using you" they finally escalated it, and then "It's our DDOS team doing, we have no bearing on them"
6. So, you see it's their official policy to bullshit clients. They clearly knew something from the start, but tried every diversion, thinking they are talking to some $10 per hour anykey man.
Soon found out that Azure silently blocks a big chunk of third world ISPs, and other small datacentre providers without an option to appeal.
> Soon found out that Azure silently blocks a big chunk of third world ISPs
Maybe the best recourse here is to get rid of the spam coming out of this ASN.
Following this logic, the first whom they need to block should be American ISPs.
North America is the biggest source of spam mail, and DDOS by far.
There's a signal to noise ratio to consider here.
This essentially echos my experience anytime I have had to deal with Microsoft support. It's unfathomable how bad their support is and the difficulty to get anything done even when presenting clear evidence/facts.
Microsoft doesn't cater to the more knowledgeable admin population I guess. They are used to the point-and-click crowd mindlessly accepting every answer as gospel. Because usually you can't prove them wrong anyways, everything being opaque and closed in Microsoft-land. And because of that, MS/Windows admins never learned how to get to the very bottom of things.
> We named this quartet of zero-days “OMIGOD” because that was our reaction when we discovered them.
i gather from this that maybe microsoft isn't the best supply chain for linux images. i find this revelation to be startling.
"When customers set up a Linux virtual machine in their cloud"
Yea, make sure to mention Linux in relation to vulnerabilities. Then pretend Microsoft is to the rescue with OMI. A FUD piece for Microsoft masquerding as a technical article.
You're saying this is promoting Microsoft? I would consider an article pointing out a vulnerability in Azure to be bad publicity for Microsoft. The title of the article also almost makes it sound like Microsoft is behaving nefariously by having a secret agent:
>“Secret” Agent Exposes Azure Customers To Unauthorized Code Execution
I think it was the multiple references to OSS being dangerous (never figured out why) left a bad taste for me.
But the vulnerability is only for azure Linux hosts so not sure how to report it without mentioning Linux.
What’s comical though, is I wonder what they use for Windows/non-Linux hosts. Is there an OMI equivalent for those hosts? Unless it’s open source, I guess we won’t know if it has a vulnerability.