"Secret" Agent Exposes Azure Customers to Unauthorized Code Execution

277 points20
MrStonedOne18 hours ago

> Open source is a supply chain risk Given the potential impact of OMIGOD, the million-dollar question is “How does an open source component with only 20 contributors on GitHub affect the security of large portions of Azure?” The ease of exploitation and the simplicity of the vulnerabilities makes us wonder if the OMI project is mature enough to be used so widely.

> Yet this scenario is more common than you might think and certainly not unique to Microsoft. One of the benefits of open source is that it’s easy for developers to grab code from different projects and mix it with other open source and proprietary software. As a result, bad open source code can wind up in an enormous range of products and services – inadvertently becoming a “single point of failure.”

The fear mongering about open source in the article seems misplaced.

Any software product could lead to the same thing, and OMI was used as a software product in this instance. I don't see how it being open source changes anything and the article doesn't address this, only goes on a tangent about how "open source code can end up in any software project".

It also leaded with a bit about supply chain attacks, in a way that seemed to imply, without explicitly stating, an intentional cause to this exploit existing.

> Because customers don’t know what franken-code is running in the background of the services they use, they remain at risk and unaware.

Wouldn't this be worse with closed source software thou?

defaulty18 hours ago

Ironically, the OMI 'open source' package is Microsoft's.

The only recent commit was for "Enhanced security" a month ago:

jahewson17 hours ago

Indeed! And most of the 20 contributors work for Microsoft. Talk about FUD.

blibble16 hours ago

I read all of this:

still don't understand what it does

q3k8 hours ago

It’s part of dumpster fire that is the DMTF ecosystem. Attempting to understand it if you haven’t been exposed to Windows enterprise bullshit is futile. It’s layers upon layers of meta XML to do the most mundane things in the most overcomplicated way without actually succeeding at making any usable standard.

Or, in other words: if you reaction to SOAP was “damn, this is amazing, we need more of this but with even more indirection”, you’ll feel right at home.

mook13 hours ago

Ars Technica had an article on three vulnerability that was decently clear:

(Sounds like it's a Linux port of some Windows management feature.)

hughrr12 hours ago

This is one reason I stay the hell away from Windows focused software on Linux. There’s a complete impedance mismatch. Also MSFT’s GitHub open source fairy dust project stewardship is a shit show from experience.

matja6 hours ago

yes, checking the secret before setting the uid/gid does seem like "Enhanced security" :

paxys18 hours ago

Exactly. It's not like Microsoft picked up a random unaudited open source project and shoved it into all Azure VMs. OMI is maintained by Microsoft and the fact that it is open source is a good thing security wise. I'd wager being able to look at the source code greatly helped the researchers in their diagnosis as well.

craigkerstiens18 hours ago

At some level, agreed that it probably helped security wise for researchers. That the fix was committed in public and then vulnerability remained for a month and well still currently for customers is where one could make a case that the open source nature did as much damage as good.

hannob9 hours ago

> That the fix was committed in public and then vulnerability remained for a month

That's an indication that Microsoft is bad at handling security issues. Which isn't news and has very little to do with open source. Something very similar happened with Exchange, it led to plenty of people being compromised, and Exchange is not open source.

formerly_proven12 hours ago

When you distribute patches for a vulnerability it basically makes no difference whether the source for those patches is available - people are quickly going to figure out what the vuln is. This is a patch rollout problem and nothing else.

zoobab8 hours ago

$ apt update

Oh, wait, fix not available in the Debian/Ubuntu official repos, only M$ trusted ones.

Veserv15 hours ago

Indeed, it is quite ridiculous to lay the blame for this on an open source project. They did not force Microsoft to use their software and deploy it to all their customers. It is entirely Microsoft's decision to accept software that waived all warranty, liability, and guarantee of fitness for purpose and then deploy it to all of their paying customers without doing appropriate due diligence.

Any company in any other field that accepted dependencies without doing quality checks to verify that those dependencies met their requirements would rightly be called grossly incompetent. This is one step further than even that as the dependencies openly guaranteed nothing and they still did not verify if it was of acceptable quality before making vast amounts of things depend on it as a critical component. This is the absolute basics of the basics and Microsoft has completely failed at even that. This is indicative of deeply-rooted gross systemic incompetence in much the same way that failing FizzBuzz demonstrates a complete lack of programming ability. As such, their opinion on matters of security and their ability to deliver security should be completely disregarded until their organization is completely overhauled.

virgilp15 hours ago

What? This is Microsoft's open source project - they didn't need to "do quality checks", they developed it!

hippyup15 hours ago

Your comment makes no sense to me - you're saying we shouldn't blame open source but in the same breath say that anyone who uses open source without carefully reviewing and auditing every line of code is an idiot who can't even do FizzBuzz. I'm a developer and if I'm faced with the choice of auditing every line of code in a dependency or just writing what I need I'll just write what I need nine times out of ten. Honestly your position is closer to the article than you seem to realize.

matt12345678918 hours ago

Yes. The explanation given under that rather provocative headline does not mention how the OMI project being open source, as opposed to closed source, enhances its status as a supply chain risk. Microsoft could have just as easily not released that code - it still would have included the vulnerability.

craigkerstiens18 hours ago

Open source supply chain is a real thing, but not sure it's at all relevant in this case. In this case it appears to be an open source package that was simply open sourced by Microsoft as opposed to some broader community project.

That it's open source there could an interesting argument made that it's easier for one to hunt for vulnerabilities when they can review the code. In this case what was the advantage or point of it being open sourced when it is clearly a Microsoft project for Microsoft purposes. Maybe for collaboration internally? But the open source nature of it doesn't gain much.

The open source nature–in this particular case becomes more complicated when the fix is in plain sight for a month now with a seemingly easy ability to take advantage of. So in this case closed source fix that wasn't public how to exploit maybe would have been better indeed.

The open source vs. not is a bit more of a red herring relative to the overall situation. It's certainly not the central theme, but when you overanalyze I can see how some of the analysis maybe happened.

devwastaken14 hours ago

The fear mongering of open source is fabricated by the very people whom intentionally include vulnerable code in proprietary software running on every device. They don't want you to see it.

beckman4663 hours ago

> As a result, bad open source code can wind up in an enormous range of products and services – inadvertently becoming a “single point of failure.”

What's the alternative? A black-box proprietary bug that becomes an NSO or NSA exploit?

The logic on this is so bad.

Linus's law: "given enough eyeballs, all bugs are shallow".

tempfs18 hours ago

Exactly, they were doing fine until they made a point to shit on all open source software instead of limiting their focus to this one, Microsoft-sponsored outfit.

randomfool14 hours ago

"Currently, this project is not accepting contributions or pull requests."

Last changed Aug 17, 2016.

pabs314 hours ago

Looks like they do use pull requests though, I guess only ones from Microsoft employees are accepted.

belltaco16 hours ago

There are also people who think open source is the holy grail of security because "given enough eyeballs, all bugs are shallow". And is used as an argument. So it cuts both ways.

prepend9 hours ago

"given enough eyeballs, all bugs are shallow" Is certainly true, but that doesn’t make OSS the holy grail.

In fact, I’ve never met anyone who thinks that. And “all bugs are shallow” doesn’t mean there are no bugs, it just means it’s easier to find bugs.

tcoff9115 hours ago

The thing about dependencies is most people don't actually read the source. But at least you have the option.

jamesfinlayson14 hours ago

Yeah I thought the list of maintainers for bash and curl were pretty short (like one or two people each).

twirlock59 minutes ago

It's hilarious we're using the actions of microsoft as an example of opensource failing.

orf18 hours ago

The apparent fix:

Looks like a standard, absolutely gigantic C/C++ project that re-invents absolutely everything including an event loop, http server, XML parser, a C++ tool to generate C code to create a “str hash” implementation, a custom lexer for WQL and a parser for a “mof” file format.

Still not clear on what it does. Seems safe.

dralley16 hours ago

> a standard, absolutely gigantic C/C++ project that re-invents absolutely everything including an event loop, http server, XML parser, a C++ tool to generate C code to create a “str hash” implementation, a custom lexer for WQL and a parser for a “mof” file format.

This is the part that gets missed when people talk about "dependency inflation" re: Rust. Absolutely, it is a problem, but most C codebases of a sufficient size and vintage are vendoring some absolutely insane hidden "dependencies" that, on average, probably get a lot less testing and attention than the average package on

There are definitely risks with both approaches.

capableweb10 hours ago

Not sure why what you're saying would be more true for C/Rust/JavaScript or any other language. The perverse idea of adding dependencies without actually reading through all the code you pull in exists in basically all language ecosystems, Rust included.

jstanley9 hours ago

In C it is much more inconvenient to include external dependencies, so it's much more common for people to roll their own implementations rather than using a tried-and-tested one.

xmaayy5 hours ago

Or copy in a .C/.h file that gets forgotten about, and therefore gets no security updates.

gsam14 hours ago

CIM/WBEM goes all the way back to 1996. They essentially wanted a management infrastructure on all kinds of devices (including different architectures, so actually C made sense then), but that also notably included remote access. At the time, SOAP was still popular, so here we are with a rather silly transport protocol and all kinds of overhead reinventing things like SSH. However, the overall goal still makes sense, it was essentially a way of 'object'-ifying everything from logs to other metrics. This fit in with the overall mode of thinking in MS with DCOM and COM (and registry), and structured configuration/management. I'm sure it's paid massive dividends on Azure Linux infrastructure. For highly structured objects, SOAP and XML aren't a terrible fit, but I doubt many people would do the same thing again today.

Honestly, they just needed to rewrite it in a safer stack. However, that still may not have saved them from all these vulnerabilities, given the scope of what they're implementing as remote management protocols. The relative scrutiny, fuzzing and manpower just hasn't been there, especially when it's obfuscated by various layers.

onionisafruit13 hours ago

Not to take away from the rest of what you said, but I don’t think SOAP was _still_ popular in 1996. I don’t think it had become popular yet. I don’t think I even heard of SOAP before 1999 or 2000. I’m not a trend setter or anything, but if it was popular, I probably would have at least heard of it.

homarp12 hours ago
jerf15 hours ago

Greenspun's law, "Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp." is really just a special case of a general principle. There's a lot of cases where if you use the wrong tool and/or don't know the right libraries, or don't or can't use them, or just won't, that you'll reinvent something that already exists, badly.

the-dude7 hours ago
Cyph0n16 hours ago

Looks like it’s mostly C, and not great C at that… In my humble opinion, this is a scary looking project.

randomfool14 hours ago

Specifically- it appears to be mostly C99, lots of macros, lots of ugly buffer manipulation. But it's tooled for PREFast so you know it's secure! ( (j/k... I haven't seen that in ages). Guessing a port of MS's internal WMI codebase?

This codebase just smells of rot all over.

pjmlp10 hours ago

That is why there is "Modern C", "Modern C++" as talked at conferences, blog posts and endless over here, and then real life code that we actually find out in production.

int_19h9 hours ago
silon429 hours ago
pjmlp10 hours ago

In typical Microsoft fashion, regardless of how much victories .NET and now Rust dev units achieve, the OS ones keep pushing for C and C++ no matter what.

That is also why we now have WinRT, WinUI, Sphere OS, Azure RTOS, regardless of what Microsoft Security advises.

jeffbee16 hours ago

The "fix" doesn't include any tests that would indicate that it actually works. Doesn't seem like a very healthy project.

rajamaka15 hours ago

The explanatory commit message of "Enhanced security" should be enough to document all you need to know!

cube0014 hours ago

You'd think a mention of the CVEs involved is mandatory for commits like these.

reilly300014 hours ago

After the JEDI black eye and slow attrition of frustrated developers/CIOs, this feels like a huge blow. Cloud is all about trust. They broke trust by installing the agent without disclosures, and shattered it with this vulnerability. Their customers running Linux vms have a pretty simple migration path to other cloud/hosting vendors, and most people have learned that only having one approved vendor for cloud just doesn’t work. Q4 and 2022 could be real rough for MSFT at least within the scope of affected customers. Moreover the torture that is Teams has got to have consequences as millions have had to suffer, and Windows 11 is a dumpster waiting to catch fire. My Threadripper 1950X can’t run Win11, and Edge is so difficult to detangle it’s practically antitrust-worthy. Oh, and setting up hundreds of thousands of developers to unknowingly commit licensing violations via Copilot…

Whatever magic Nadella brought to revive the company is starting to wear off, at least from where I’m sitting.

908B64B1971 hour ago

> JEDI black eye

How much of that is due to the new administration not liking the previous one?

> slow attrition of frustrated developers/CIOs

Citation Needed

> Moreover the torture that is Teams has got to have consequences as millions have had to suffer

Then why not use one of the 10 competitors for chat apps?

isoprophlex12 hours ago

This can't be real, I thought. This is just a stub in a random commit somewhere.

But no. Same code is on master branch atm.

What is this, a joke?

"Never attribute to humor, that which is adequately explained by incompetence"

dspillett9 hours ago

I'm going to give some benefit of the doubt and assume¹ that is some sort of stub for dev/test, replaced by one of a selection of proper symmetric encryption options in any production use. Amusing anyway.

¹ Anyone with more spare time than I want to look deeper to confirm or contradict?

formerly_proven8 hours ago

It's used there:

Also note that neither EncryptData nor DecryptData have any way of passing a key in their API. So it's very unlikely that these implementations could be conditionally compiled in place of a real implementation (but they're the only definition of these functions in the repository anyway).

mulander8 hours ago

It used to have actual Windows specific crypto code[1] which has been removed in the linked commit.

I assume this has been ported from Windows and later never implemented the ripped out components. That said, I don't know the windows API so apart from confirming that they exist in Windows docs[2] I can't assess how valid their usage was.

[1] -

[2] -

dspillett7 hours ago

> neither EncryptData nor DecryptData have any way of passing a key

There could have been other smells involved, like the key being held in some more global scope instead of being passed in via the call stack.

cube0014 hours ago

Let's hope memcpy hasn't been #defined

watermelon015 hours ago

At least it's fast. :D

jamesfinlayson14 hours ago

And properly SAL annotated too.

haimez14 hours ago

An encryption scheme with truly memcpy-like performance characteristics.

asien11 hours ago

No surprise to be honest.

What rather surprises me are the comments claiming MSFT is suddenly going to go bankrupt because there is a pre-install spyware.

Have you ever worked in a Fortune 500 ?

Do you know how hard it is in those company to get anything done that has not been budgeted and planned years in advance ?

Do you seriously believe Fortune 500 using MSFT are going to suddenly migrate ALL their Azure workload somewhere else like some kind of startups run by 3 devs ?

Yes Azure VMs have spywares builtin , but last time I checked all Europeans companies that are using American Cloud fall under the « Cloud Act » legislation which « legally » requires cloud vendor to hand over ALL the data the company is currently hosting.

I’ve worked in some of the largest insurances in Europe they LOVE Microsoft and Azure , even if there is this spyware issue , they will pretend it doesn’t exist and do business as usual.

Im pretty sure everyone will forget about this news in 1 month or so.

smcleod17 hours ago

It blows my mind how easy it is to get root with this - "Simply remove the auth header and you are root."

reilly300014 hours ago

That the kind of thing you get with homegrown web servers and corporate deadlines. I see how a QA person wouldn’t test that case, it’s almost a base assumption that the header would be mandatory.

icecap1217 hours ago

Wiz is definitely trying to drum up some excitement around their new business with all these recent disclosures. Related but slightly off topic, we pay an industry analyst for high level market research and he's been screaming from the building tops about for the last 6 months. I'm honestly tired of hearing about it, he must get a cut of the sale. I wonder how long Wiz will keep this up; until they get sold presumably, then they'll go the way of the other big cyber firms and tone down the big hacks.

shir115 hours ago

If you want to see what all the hype is about - I can try arrange a demo meeting for you ;)

bostik13 hours ago

Peeking in from the sidelines, as someone who probably will want to arrange a demo too, here's a thing I've been scratching my head with. Your online pitch contains a logical puzzle I haven't been able to decipher.

Patch management and vulnerability checks, right. (VM) OS configuration checks, sure, why not. "No agents or sidecars to deploy" ... hang on, how does that combine with the other two? In order to check for installed/missing patches for on-system software you have to have some kind of access to the underlying system.

Surely you are not snapshotting root volumes for your analysis needs?

flyinglizard16 hours ago

Very slim chance it’ll get sold. It’s going for an IPO to play alongside other cyber security companies like Sentinel One. The founders are already rich enough from previous ventures.

DrAwesome18 hours ago

While I think this article is unnecessarily critical of the Azure OMI agent, this is a very "What the heck, Microsoft!?" moment for me. Of all the pieces of Azure infrastructure, the OMI agent is absolutely something I expect to be well-tested and secure.

I recognize that bugs happen, but allowing a remote client to execute commands as root by simply removing the authorization header should have been caught by automated testing.

thesuperbigfrog2 hours ago

This announcement seems badly timed given the OMIGOD vulnerability:

"Microsoft announces passwordless future – available across Microsoft Edge and Microsoft 365 apps":

AnimalMuppet2 hours ago

No, see, it's totally consistent. Instead of just hackers having passwordless access, now everybody can!

/s, if it wasn't obvious...

azurezyq13 hours ago

GCE's counterpart doesn't seem to have a public endpoint and its functionality seems make sense:

I have to say the problem is not oss, not agents, but Microsoft.

throwdbaaway7 hours ago - I think this is the equivalence of GCP guest-agent, serving similar functionalities, and is pre-installed on all official images, otherwise basic things like authentication and image baking will break.

By setting the provisionVMAgent property to false when creating a virtual machine, WALinuxAgent should run with all extensions disabled, and I think that's as minimal as a Linux VM can go on Azure.

This property, however, can't be set via, which is of course another lovely OSS project by Microsoft. I didn't bother to send a PR.

The OMI agent seems to be a different beast that is way more obnoxious. The closest thing on GCP is probably the collectd agent and the fluentd agent installed for Stackdriver Monitoring and Stackdriver Logging? Plus whatever OS config to enable unattended upgrades.

I just learnt from this HN thread about the SSM agent on AWS. That one does seem equally obnoxious as the OMI agent.

EDIT: Looks like collectd and fluentd have been replaced by

dialogbox11 hours ago

And it is written in Go. I think it is generally a safer choice.

throwdbaaway7 hours ago - it used to be just one project with Python, C++, and shell scripts. The Python bits got rewritten into Go, while the others got split into their own repositories.

throwaway98439318 hours ago

Wait, is Azure not patching this on customer systems? They're leaving it up to customers to patch a hole they themselves introduced?

reilly300014 hours ago

That immediately makes me wonder about AWS SSM, which I spotted preinstalled on a Ubuntu 18.04 AMI the other day. I think it can’t be accessed outside of the VPC but it’s worth further research. Agents are scary.

snvzz10 hours ago

> open source component with only 20 contributors

How's that not a lot of people? I don't get the "only" at all.

Is this meant to be a joke?

prepend9 hours ago

I was surprised by this as well. 20 people is a lot for a project, especially an open source project.

I would have liked to see what the author thinks is a sufficient number of contributors.

I’m pretty cynical because I’ve frequently seen a lot of pot calling the kettle black accusations when the authors projects are shown to only have 1 or 2 contributors.

flyinglizard18 hours ago

This is the second Azure disclosure from the guys at Wiz, following the one with Cosmos DB few weeks ago. Now, it’s much more interesting when you take into account that Wiz was founded by the guys who sold a cyber security company, Adallom, to Microsoft and then served in senior roles around Microsoft cloud security post acquisition. Assaf Rappaport, Wiz CEO, served as fhe GM of the Cloud Security Group at Microsoft for five years [0].

I wonder what the people at Microsoft are thinking of this situation.


jcims17 hours ago

I'm sure its awkward but they appear to be an equal opportunity reporter -

Their product is pretty interesting. I did a demo a while back and their approach of building a large graph and doing a good amount of reachability (network and entitelement) leads to some useful signal. It's not entirely novel, JupiterOne has been doing something in the same vein for years now but it seems to work well. Still feels a bit rough around the edges but i thought it was interesting and could work well in situations where simple checkbox-style config analysis falls short.

keiko-z7 hours ago

I was able to leverage JupiterOne to identify the misconfiguration mentioned in the Wiz article across all my AWS accounts in a single query. Pretty nifty.

They shared that query and a bunch more in this blog:

kerng17 hours ago

Interesting. Would be curious to know what parts they worked at. GM seems high enough to know about "areas to look at".

Still doesn't excuse having such bugs in the first place though.

paulryanrogers15 hours ago

What's the implication? That faults were included or neglected to be sussed out by their next company?

flyinglizard7 hours ago

Absolutely not. But it’s still kinda weird. Like Facebooks CISO leaving their post to found a company which promotes itself with Facebook CVEs. To quote Matt Levine, “I don’t know”.

lloydgrossman17 hours ago

> Now, it’s much more interesting when you take into account that Wiz was founded by the guys who sold a cyber security company

security people sell security company and create a security company. color me shocked.

yellow_lead15 hours ago

Nice out of context quote.

baybal216 hours ago

I am using Azure on one project as per client's requirement.

One thing I can say, Microsoft hasn't changed a bit. It's a Win98 experience in the cloud.

Now we are fighting Microsoft silently blocking entire ASN of Airtel in Nigeria, and:

1. First, obviously pretending having no idea what are we talking about, and bullshitting us

2. Then not acknowledging. "We checked it, everything is fine"

3. When faced with traceroute - "Maybe it's a third party network provider"

4. When faced with WHOIS record - "I don't know what really to do"

5. "When faced with "this is the last day we are using you" they finally escalated it, and then "It's our DDOS team doing, we have no bearing on them"

6. So, you see it's their official policy to bullshit clients. They clearly knew something from the start, but tried every diversion, thinking they are talking to some $10 per hour anykey man.

Soon found out that Azure silently blocks a big chunk of third world ISPs, and other small datacentre providers without an option to appeal.

908B64B1971 hour ago

> Soon found out that Azure silently blocks a big chunk of third world ISPs

Maybe the best recourse here is to get rid of the spam coming out of this ASN.

baybal21 hour ago

Following this logic, the first whom they need to block should be American ISPs.

North America is the biggest source of spam mail, and DDOS by far.

908B64B1971 hour ago

There's a signal to noise ratio to consider here.

gibs0ns14 hours ago

This essentially echos my experience anytime I have had to deal with Microsoft support. It's unfathomable how bad their support is and the difficulty to get anything done even when presenting clear evidence/facts.

corty9 hours ago

Microsoft doesn't cater to the more knowledgeable admin population I guess. They are used to the point-and-click crowd mindlessly accepting every answer as gospel. Because usually you can't prove them wrong anyways, everything being opaque and closed in Microsoft-land. And because of that, MS/Windows admins never learned how to get to the very bottom of things.

tomcam18 hours ago

> We named this quartet of zero-days “OMIGOD” because that was our reaction when we discovered them.


a-dub9 hours ago

i gather from this that maybe microsoft isn't the best supply chain for linux images. i find this revelation to be startling.

duckhelmet16 hours ago

"When customers set up a Linux virtual machine in their cloud"

Yea, make sure to mention Linux in relation to vulnerabilities. Then pretend Microsoft is to the rescue with OMI. A FUD piece for Microsoft masquerding as a technical article.

Thorrez16 hours ago

You're saying this is promoting Microsoft? I would consider an article pointing out a vulnerability in Azure to be bad publicity for Microsoft. The title of the article also almost makes it sound like Microsoft is behaving nefariously by having a secret agent:

>“Secret” Agent Exposes Azure Customers To Unauthorized Code Execution

prepend8 hours ago

I think it was the multiple references to OSS being dangerous (never figured out why) left a bad taste for me.

But the vulnerability is only for azure Linux hosts so not sure how to report it without mentioning Linux.

What’s comical though, is I wonder what they use for Windows/non-Linux hosts. Is there an OMI equivalent for those hosts? Unless it’s open source, I guess we won’t know if it has a vulnerability.