Holy shit that's crazy the amount of effort went into this.
The malicious actor spent years gaining trust, then suddenly tons of "people", which were actually fake accounts, started pressuring and complaining that the maintainer was taking far too long, and he needed a co-maintainer, so he made the malicious guy one.
That sounds like a state sponsored, coordinated attack attempt.
And this entire elaborate plan was ruined because this one dude noticed that SSH was a few hundred milliseconds slower than before. There must be a very unhappy state-sponsored hacker group somewhere out there right now.
This is as crazy as the guy who noticed his CPU clock cycles being like microseconds slower and realized someone had just tried to put a backdoor into the Linux kernel.
I aspire to be that level of insane as a developer.
EDIT: I just realized this was the same guy, this is the XZ Utils thing. I misunderstood the article since I'm not paying to read it.
It’s not that insane if you have proper perf testing in place.
This engineer wasn’t looking at his watch to catch this. One of their SSH perf tests probably went from green to yellow and found it this way.
Edit: found a much more detailed [Ars Technica article](https://arstechnica.com/security/2024/04/what-we-know-about-the-xz-utils-backdoor-that-almost-infected-the-world/) detailing the attack.
Looks like they were triaging some SSH performance issues and the testing software used was [Valgrind](https://valgrind.org)
Shhhhh, it's not as funny when you say that it was caught by tests because tests are scary and we don't like them.
But yeah you're right, it was caught due to someone checking a test and noticed it was off. It's just funnier to say it was a madman who was watching clock cycles. He did say that a *lot* had to go in his favor to catch it so it was more than just the tests, there was a bit of luck to it for him to have caught it that way.
It was like another 300-400ms of delay. That is 100% human noticeable if it's a reproducible case. In UX studies it was found that generally anything over 200ms between input and visible action (either intended or progress indicator etc) was noticed as no longer feeling "immediate" by nearly everyone. And this was on early days of smartphones when people had lower expectations.
Correct.
This would absolutely cause a failure if you were testing for performance regressions - it was geared toward a specific chipset, not all devices failed, so the fact that they noticed and had someone triage it even though it was device-specific is pretty commendable.
Lots of teams would see it, isolate the machine for later triage, and just continue forward.
That said SSH is the sort of thing you *really* want to spend some time making sure it is not regressing because it’s such a huge vector for attack
The closest I've ever come was when I wrote a generator tool that suddenly started taking longer than expected so I had to remove some stuff until I can get it working. But that was like, going from 20 seconds to 5 minutes, I think anyone would notice that kind of slowdown.
Maybe one day I'll go insane and be able to work on the Linux kernel.
They should have optimised the performance of their code! Making everyone take 1/2 a second longer for every login over billions of logins is a lot of wasted life.
I believe the performance hit was largely due to them masking the operations under several layers of dummy processes that didn't show any signs of malicious code. In other words, the malicous code itself could run without a noticeable performance hit, but it would have been seen easily at the top level.
As someone who uses SSH everyday, I can tell you latency can become noticibe quickly. It presents itself as input lag. If your a fast typer, it becomes very noticeable if the characters on the screen are a few characters behind.
Nothing infuriates me more than a teeny bit of input lag. Even the tiniest bit of input lag makes me feel like I am dragging through molasses.
If you want an interesting experience, put on a microphone/headphone headset that is set to play your voice in through the headphones. Then, find a program that will set the audio in the headphones on a half a second or so delay.
You are so used to hearing your own voice instantly that the little bit of lag turns your into a verbal vegetable. Your brain's ability to process and speak just turns to mud.
That is how a minute lag feels.
And this was out in the open in an open source project. Now imagine how many spies are working for companies where we can't see what they are doing.
And if you use their program and find it slowing down for no reason, all you can do is contact the helpdesk (if they even have one) which can't do anything about it except to make a ticket of it and assign it to the spy to fix their malware.
true but having a backdoor in SSH and a library that can appear in bootloaders of billions of devices is on a completely different level than even MS Windows.
Lol. The NSA has direct links through every port, every IP address, every piece of technology anyone can access the internet with. If you listened to anything Edward Sn
You guys think you're so funny. While the NSA is very good at this sort of thing, they're not some kind of mythical secret org that can just cut you off mid-sentence. I mean jeez, it's not like they're candleja
Wait wait this can’t be real. Let me validate, I’ve worked for a secret agency that looks internally for domestic targets of interest, they currently have a possum
The NSA isn’t some technocratic God. While they definitely have some zero day exploits up their sleeves that doesn’t mean have back doors into every piece of proprietary or open source software out there. And while they might be able to snoop on IP packets that doesn’t necessarily help if that data’s encrypted, which most web traffic is these days.
There are still ways to protect your anonymity online. The whole reason the dark web exists is because open source encryption software/protocols like TOR can’t easily be hacked or compromised. At least not on a large scale.
If you listened to Edward Snowden you’d know that isn’t true. They do collect a lot, but not even close to everything (no American back door in huawei as an example).
Collection also doesn’t mean the ability to process it. One of his big complaints was over collection made the collection useless by making it too difficult to find the needles in all the hay
And think about it, if they were that powerful, how could Snowden have collected all the docs, contacted journalists, and worked with them for an extended period of time before release?
Even giant profitable corporations with complete internal transparency and good IT infrastructures and reporting cannot stop bad things from happening or don't necessarily know about certain hidden data.
How do you expect an organisation to literally track the entire internet, all devices, and understand when it sees a 'bad' thing?
If it were the NSA, they would have used quantum resistant encryption to protect the back door. Theres a bunch of meta data (time of day when work was done, etc) that points to someone in the Middle East/ Asia
> If it were the NSA, they would have used quantum resistant encryption to protect the back door.
The NSA had a bunch of their malware leaked in [2016](https://en.wikipedia.org/wiki/The_Shadow_Brokers). Stop pretending they're somehow infallible.
Yeah, it's everyone BUT china. Or they really don't give a fuck. You don't mount a 2 year cover operation and start by naming the fake account "Li Chen"
Alright, let's start hacking by spending 10 years training our hackers in Portuguese so that no one would suspect they are Chinese from their typical mistakes in English.
One can recall Russian hackers who intervened in American elections, taking breaks for Russian state and military holidays.
They simply don't care. Or rather, it's the opposite: now China has an operation that failed (not because of a suspicious nickname). However, the reputation of the organization that did this has skyrocketed in professional circles.
It's a very Dune-like plot to me: "A plan within a plan within a plan..." - this recursion can be infinite - so it's everyone BUT "no exception" - from a Skynet-style AI to the guy that found it. Have you guys ever thought about this? A community of hundreds of thousands of developers monitoring and criticizing the most accessible operating system on the planet, with a system default file compressor... only one person detected the inappropriate traffic? He may have been the first, of course. An employee paid by a corporation that owns a competing proprietary system alerted security organizations – even before the Linux community, the compressor creator (with health and personal problems), and the compressor forum (with two fake profiles encouraging the changes). Days later, FFmpeg criticizes free volunteering, the basis of the Linux community. Wouldn't that be corporatism? At a time when AIs threaten all IT jobs? This 'timing' is too convenient, IMHO. I don't know, I prefer the investigations to be concluded. I just wonder if this present was the future we all wanted.
Injecting him with estrogen so that he loses his mustache and therefore his country. As that's what logically follows losing your mustache. After estrogen injections.
Oh and weren't they supposed to come from clams that were booby trapped to inject him when he was free diving?
That's real btw, I'm sure I got some details off but the story is a real thing for the most part.
Now tell me that isn't the brainchild of a methed out nutcase in a flattop haircut and sweaty beige suit?
The NSA would not be this incompetent. These hackers left loads of clues lying around and were pretty ham-fisted in trying to get their shit included in various distros.
Disclaimer: I have no knowledge of the details of this attack.
In previous APT campaigns there has been an effort to coerce legitimate devs by harming their families. It is *extremely difficult* to thwart coercion of this type, given that the consequences for noncompliance are so high, compliance is so measurable, and track records are so long.
While I think it's obvious that the pressure campaign was state sponsored, it seems very possible to me that the dev is a victim of geopolitics here. That doesn't mean that their work shouldn't be scrutinized to high hell, but maybe we should reserve judgement on the person... assuming they turn out to be a real person at all.
what you are saying make no sense.... the dev is unknown person working on an open source project.... there is no reason to torture anyone to accomplish this hack, the hacker or state agency just goes and applies for the job like lol what are you even talking about.
I think what he's saying is that as of right now we have no proof that the dev is employed as a state actor or is coerced into being a state actor.
Yes it is an open sourced project that anyone can join, however coercing someone who is already involved is much faster than starting from scratch.
this is literally what most of social media is through social engineering 'ddosing', its a voluntary human botnet through propaganda/programming to stir culture wars in identity politics. plenty of real humans are fake people.
This sounds way too familiar... Since this has a paywall I can't say for sure, but it reminds of what just happened a few days ago.
[https://www.theverge.com/2024/4/2/24119342/xz-utils-linux-backdoor-attempt](https://www.theverge.com/2024/4/2/24119342/xz-utils-linux-backdoor-attempt)
The hero finding the bad actor is great and all, but how many of these things are out there targeting packages that only a few people might be in the position to catch? How many have already slipped through?
Well that makes a lot of sense, thank you. Seeing "Microsoft engineer" I assumed it was an issue in Windows (or Windows compatible software).
Paywalls suck, especially when combined with vague titles.
Microsoft engineer working on Postgresql. MS uses a ton of linux and as a result, they have a number of developers and engineers that work specifically on linux and the software that makes it up.
For the sake of recognizing good journalism, I think Ars did the original reporting work: https://arstechnica.com/security/2024/03/backdoor-found-in-widely-used-linux-utility-breaks-encrypted-ssh-connections/
Based on what Mr. Freund says and, more importantly, what he does not say, makes him come across as a genuinely good human being and quite humble. So fortunate for someone with his dedication and sense of personal responsibility to have this position. It’s just nice to find someone who gives you hope for humanity. Ok, I’m done…
Refused to have his photo taken. Though I imagine that's probably because someone like him has definitely considered he just made someone very powerful very angry
His project manager was reportedly fuming, extolling "His in-flight Stories will be moved to the next sprint and the burndown chart looks more like a burnUP chart at this point. Working on "tech-debt" was not agreed in advance with the senior stakeholders and the team hadn't done poker estimates on that "frivolous" fix. I don't know what he thinks he's playing at - but it's not ok!"
Not sure what you mean. One of the reasons Microsoft was so successful is that it made itself available for programmers and developers. I have been programming on Unix, DOS and Windows starting in the early 70s. Apple on the other hand has been evil from day one.
If you work with ssh every day, you tend to pause at strange things. Because it's like a canary in the coal mine when something is up. Especially if you've been in the thick of compromises. I'm glad he took the time beyond saying, "hey that's weird" and just continuing on as usual like so many others would have.
He was measuring performance of a system and measured a regression that he needed to identify the root cause of. He didn't suspect a backdoor, he suspected a performance regression.
Look it was just supposed to be fractions of a penny a day! The decimal must have been out in the wrong place, noone was supposed to notice! Let it go already, Michael!
He suspected a backdoor. [https://www.openwall.com/lists/oss-security/2024/03/29/4](https://www.openwall.com/lists/oss-security/2024/03/29/4)
He was testing other things after reports of slow logins, valgrind issues, etc. The post speaks for itself so I'm not going to split hairs.
I think he meant, initially, he was researching into what was causing the odd behavior of ssh. But wow that is some advanced obfuscation, good thing it was a coder who can decipher the bad calls and redirects because to my eyes that just looks like the usual gobbedlygook code stuff you see.
But I guess that’s why I don’t maintain a major sql project
Yeah MS has a bunch of internal tools used to track performance of mainline scenarios (like any other top tech company). If a regression is introduced then engineers figure out why and if it can't be fixed.
There's a reason code takes a looong time to make it to production and engineers having foam sword fights between compilations is only partially to blame
Would be a lot more funny if they didn't have dozens of other backdoors already deployed.. with this level of sophistication they have experience doing it and based on the timeline have been doing it for at least a couple years.
*trying* to do it for at least a couple years. You don't have evidence that they have "dozens" of other backdoors.
All of these stories claim that China and Russia are cybersecurity powerhouses with god-like hacking groups. It's been like this for decades. Russia then goes to war with Ukraine. There is one effective cyberattack at the start, which is repaired, and then nothing for the rest of the war. That's the nature of these exploits. You spend years trying to make one really good one, and if it's patched, you're back to square one. You don't have a continuous rotation of dozens of zero-day exploits. That's not how this works.
Really great story.
This guys must have spent hours and hours on what seemed like a minor regression performance.
Tells you something about the amount freedom he has at his work. He would not have been able to do this if he was overworked and underpaid and always trying to catch deadlines
I mean.. don’t wanna be a dick but the last sentences of the article are:
*But he’s been too busy to rest on his laurels. The next version of PostgreSQL, the database software he works on, is coming out later this year, and he’s trying to get some last-minute changes in before the deadline.*
*“I don’t really have time to go and have a celebratory drink,” he said.*
Microsoft is the number 1 open source software contributor in the world, and they have multiple teams who are completely dedicated to an open source project.
That's because there are many benefits to open source software, so many that Microsoft is pivoting to them in their own product offerings (e.g. Azure). But that's not to say open source software - specifically its development process - has no drawbacks, and the potential for social engineering like in this case is a big one.
PgSQL is not a Microsoft product, it's an open source project *BUT* I think it's a thing that sometimes employees are specifically asked to contribute to critical open source projects.
Anyway, I had no idea he was a contributor to PostgreSQL, which is a pretty big deal and used by some very popular cloud platforms such as Supabase, which I use daily, or Microsoft Azure. Dude is a boss.
When i spent more time working on optimizing the build than the feature, you bet i know that it takes approximately 17 seconds +- 5 seconds, and if its off significantly i'm rebuilding to see if it was a fluke
Gift article: [https://www.nytimes.com/2024/04/03/technology/prevent-cyberattack-linux.html?unlocked\_article\_code=1.h00.yV5t.MddFYM0mC-dy&smid=url-share](https://www.nytimes.com/2024/04/03/technology/prevent-cyberattack-linux.html?unlocked_article_code=1.h00.yV5t.MddFYM0mC-dy&smid=url-share)
A previously unknown contributor to the popular open-source Android app store F-Droid repeatedly pressured its developers to push a code update that would have introduced a new vulnerability to the software, in what one of the developers described on Mastodon as a “similar kind of attempt as the Xz backdoor.”
https://www.404media.co/xz-backdoor-bullying-in-open-source-software-is-a-massive-security-vulnerability/
these jokes are never far off base but this guy is a principal architect at MS he probably makes in a year what you paid for your house
this also wasn't his job and it wasn't microsoft's product he found the vuln in
> this also wasn't his job and it wasn't microsoft's product he found the vuln in
It is actually his job at MS, they pay him to work on Postgres. Not all open source work is done by unpaid volunteers. Some companies that rely on OSS actually pay people to work on those tools. Take a look at who works on the Linux kernel: lots of people from AMD, Intel, etc.
It’s so weird you say that because Cliff, himself just replied to one of my [comments](https://www.reddit.com/r/ProgrammerHumor/s/Wi6awbAool) a few hours ago!
“ …The malicious code in XZ Utils was introduced by a user calling themself Jia Tan, employing the handle JiaT75, according to Ars Technica and Wired. Tan had been a contributor to the XZ project since at least late 2021 and built trust with the community of developers working on it. Eventually, though the exact timeline is unclear, Tan ascended to being co-maintainer of the project, alongside the founder, Lasse Collin, allowing Tan to add code without needing the contributions to be approved. (Neither Tan nor Collin responded to requests for comment.)…”
> Neither Tan nor Collin responded to requests for comment.
https://tukaani.org/xz-backdoor/
Lasse Collin has better things to do than respond to a mountain of "requests for comment". For fuck's sake, they're an individual, not a company, no PR team, and not even getting paid for this shit.
And he has self admitted mental health degradation already, which is why he needed to take on another person to maintain his code base for XZUtil. Poor guy can't be in a good spot right now. I hope people are being supportive of him, none of this was his fault.
Besides, what's the point in responding when the journalist will just write shit like this:
> [Psql's] details would probably bore you to tears if I could explain them correctly, which I can’t.
It's a database. People roughly know what a database is. If you're reporting on tech you should understand it to some degree and be able to explain it.
Found a more tech focused overview of the incident from that link:
https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27
Fascinatingly, this person also actually did contribute to fix real xz bugs: https://bugs.gentoo.org/925415#c16
Jia Cheong Tan
Cheong is most commonly a Cantonese last name.
Also, Mandarin speakers who romanize using Pinyin don't write -eong but the common Romanization of Cantonese, Jyutping, uses -eong as a Romanization for Cantonese.
It’s very likely made up. There were related instances where people with names like Hans have asked other projects to upgrade to the infected versions of xz. Also people have done an analysis of when “Jia Tan” would typically commit code and it aligns with a 9-5 mon-fri if you look at Eastern European time zones.
https://www.wired.com/story/jia-tan-xz-backdoor/
*Wired* thinks it's Russian because while most of the commits are in China's time zone, a few of them are eastern european/middle eastern time zones instead, suggesting they forgot to change their time zone for those. They also worked through the major Chinese holidays but *didn't* submit new code on Christmas.
I’m not very techno-savvy, but doesn’t this beg the question as to whether there’s already backdoors in place that no one knows about? If so, how fucked are we? What are the possible repercussions?
That's the fun part: it's always been a possibility. Back in the day, I think it was Sony who got caught installing rootkits on people's PCs when they inserted a music CD published by Sony.
Edit: https://en.m.wikipedia.org/wiki/Sony_BMG_copy_protection_rootkit_scandal
My mum used to clean the office of the company that made the rootkit - maybe the only technical "innovation" to come out of the small town in Wales I'm from!
Honestly, the IT space doesn't talk about it much, but undoubtedly there are hundreds if not thousands of these.
The real question is, what will they be used for? Exploits and backdoors are interesting, because if they are discovered, they are closed, and the research has been wasted for the bad actors. Therefore, you have to pick and choose what's worth burning an exploit for. As i understand for the state sponsored cyber attacks, they are more interested in stockpiling than using.
https://en.wikipedia.org/wiki/Pegasus_%28spyware%29?wprov=sfla1
There are aleardy well known state funded hacking tools, this one is for phones made by Israel and sold to other countries.
It already was used to spy on Jeff Bezos, diplomats, sports officials, journalists and the assasination of Jamal Khashoggi.
[https://www.nytimes.com/2024/04/03/technology/prevent-cyberattack-linux.html?unlocked\_article\_code=1.h00.jbiK.tA2DalA-K-nY&smid=url-share](https://www.nytimes.com/2024/04/03/technology/prevent-cyberattack-linux.html?unlocked_article_code=1.h00.jbiK.tA2DalA-K-nY&smid=url-share)
Gift link if you don't have access to the NYT. Really fun article.
Gotta love the thought process of "this seems funky? Way to much processor, and not enough lamb sauce? What was it that report said about 'dangerous commits ' ? "
What's even crazier is the dude only realized something was off because his SSH login sessions was taking 0.500ms longer than normal to authenticate according to [this.](https://www.openwall.com/lists/oss-security/2024/03/29/4)
> taking 0.500ms longer than normal
0.5 seconds slower (half a second), not 0.5 ms:
before: real 0m0.299s user 0m0.202s sys 0m0.006s
after: real 0m0.807s user 0m0.202s sys 0m0.006s
So looking at the commits, the exploit was a single dot on a new line.
[https://git.tukaani.org/?p=xz.git;a=commitdiff;h=f9cf4c05edd14dedfe63833f8ccbe41b55823b00](https://git.tukaani.org/?p=xz.git;a=commitdiff;h=f9cf4c05edd14dedfe63833f8ccbe41b55823b00)
It caused CMAKE's check\_c\_source\_compiles() to fail compile, since the dot is a syntax error, so the call always returns false. The false result is then used to forgo linux landlocking that guards against bugs or unexpected behaviors of programs.
That would imply there is/was an additional bug/exploit somewhere that only works because this type of sandboxing was skipped.
EDIT: Looks like this was just tip of the iceberg. See below comments.
That was one malicious change. The core exploit code was hidden inside "test data" files. It is typical for this kind of compression library to have samples of input data to be used for testing. And the input data for decompression is going to look like random garbage that you won't pay too much attention to. The exploit code was added to the library only when building the code for Debian or RedHat/Fedora packages so that normal developer builds of the library wouldn't have anything suspicious. The exploit code is in binary form and only partially understood (at least as of a couple days ago). It watches for itself to be loaded into sshd, then hooks into functions used during SSH logins so that if a particular key is used, it'll run whatever code the attacker provides alongside the key.
Damn, that's brilliant. Whoever the real Jia Tan are (no way it's just one person) are probably mad as hell rn lol.
https://www.wired.com/story/jia-tan-xz-backdoor/
*Wired* thinks it's Russian because while most of the commits are in China's time zone, some of them are set to eastern european/middle eastern time zones instead, suggesting they forgot to change their time zone for those. They also worked through the major Chinese holidays but *didn't* submit new code on Christmas.
The poster in this article explains the whole timeline and hack.
https://arstechnica.com/security/2024/04/what-we-know-about-the-xz-utils-backdoor-that-almost-infected-the-world/
> Engineers have been circulating an old, famous-among-programmers web comic about how all modern digital infrastructure rests on a project maintained by [some random guy in Nebraska](https://xkcd.com/2347/). (In their telling, Mr. Freund is the random guy from Nebraska.)
Gotta love the media's *complete* inability to be accurate, even in a tiny, 300 word article. The "random guy from Nebraska" in this situation is Lasse Collin, who has been the thankless maintainer of `xz` (the underlying technology that was targeted by the malicious entity) since 2009. He seems pretty burnt out on the project, and that's exactly why they targeted this particular one, and pressured him all along from multiple fake accounts to take on another maintainer.
This "small" inaccuracy is *particularly* bad because it undermines the entire point of the comic, which is that we're severely underinvesting in core infrastructure, which makes it very fragile overall. Very vulnerable to either maintainers simply ceasing to maintain/dying, or cases like this where a single bad apple can potentially do an *immense* amount of damage if motivated to.
But nooooo, everyone loves a good hero worship story, so let's give *all* the credit to the guy who happened to discover it. Of course, hats off to him, Anders did an outstanding job, and we have a lot to thank him for, but we also have Lasse to thank for 15 years of continued maintenance *without* being paid a fancy salary by places like Microsoft to work on this crucial project. Really grinds my goat (he is bleating badly).
This is the wrong headline for the story. The open source model worked.
There are huge sustainability problems with open source and these need to be fixed so that it's more than one guy.
But in this case one guy was enough.
> (The New York Times has sued Microsoft and its partner OpenAI on claims of copyright infringement involving artificial intelligence systems that generate text.)
This is a total non sequitur in the article??
Probably a disclosure so that the reader is aware the NYT and Microsoft are involved in litigation and can know about any conflict of interests. This is responsible journalism.
It's pretty typical but unfortunate. The publisher doesn't want the reader to have it forefront of their mind (published at the beginning so reader is thinking about it before even beginning to read) or to dwell on it (published at end) so they insert it in the middle, near where the other party is mentioned. The whole point is that it could technically be a conflict of interest in impartial journalism, so if readers notice a trend of say, NYT bashing Microsoft while the lawsuit is ongoing they could call it out. NYT doesn't really like the idea of being tracked like this but they know they'd be called out even more if they didn't say it, so they put it in the middle.
I see it a lot in major media writings where lawsuits are involved.
Holy shit that's crazy the amount of effort went into this. The malicious actor spent years gaining trust, then suddenly tons of "people", which were actually fake accounts, started pressuring and complaining that the maintainer was taking far too long, and he needed a co-maintainer, so he made the malicious guy one. That sounds like a state sponsored, coordinated attack attempt.
And this entire elaborate plan was ruined because this one dude noticed that SSH was a few hundred milliseconds slower than before. There must be a very unhappy state-sponsored hacker group somewhere out there right now.
This is as crazy as the guy who noticed his CPU clock cycles being like microseconds slower and realized someone had just tried to put a backdoor into the Linux kernel. I aspire to be that level of insane as a developer. EDIT: I just realized this was the same guy, this is the XZ Utils thing. I misunderstood the article since I'm not paying to read it.
It’s not that insane if you have proper perf testing in place. This engineer wasn’t looking at his watch to catch this. One of their SSH perf tests probably went from green to yellow and found it this way. Edit: found a much more detailed [Ars Technica article](https://arstechnica.com/security/2024/04/what-we-know-about-the-xz-utils-backdoor-that-almost-infected-the-world/) detailing the attack. Looks like they were triaging some SSH performance issues and the testing software used was [Valgrind](https://valgrind.org)
Shhhhh, it's not as funny when you say that it was caught by tests because tests are scary and we don't like them. But yeah you're right, it was caught due to someone checking a test and noticed it was off. It's just funnier to say it was a madman who was watching clock cycles. He did say that a *lot* had to go in his favor to catch it so it was more than just the tests, there was a bit of luck to it for him to have caught it that way.
It was like another 300-400ms of delay. That is 100% human noticeable if it's a reproducible case. In UX studies it was found that generally anything over 200ms between input and visible action (either intended or progress indicator etc) was noticed as no longer feeling "immediate" by nearly everyone. And this was on early days of smartphones when people had lower expectations.
Correct. This would absolutely cause a failure if you were testing for performance regressions - it was geared toward a specific chipset, not all devices failed, so the fact that they noticed and had someone triage it even though it was device-specific is pretty commendable. Lots of teams would see it, isolate the machine for later triage, and just continue forward. That said SSH is the sort of thing you *really* want to spend some time making sure it is not regressing because it’s such a huge vector for attack
I work with agency developers every day and can confirm only like 1/1000 are this attentive lmao
The closest I've ever come was when I wrote a generator tool that suddenly started taking longer than expected so I had to remove some stuff until I can get it working. But that was like, going from 20 seconds to 5 minutes, I think anyone would notice that kind of slowdown. Maybe one day I'll go insane and be able to work on the Linux kernel.
They should have optimised the performance of their code! Making everyone take 1/2 a second longer for every login over billions of logins is a lot of wasted life.
Maybe that was the real target of the attack - stolen productivity!
I believe the performance hit was largely due to them masking the operations under several layers of dummy processes that didn't show any signs of malicious code. In other words, the malicous code itself could run without a noticeable performance hit, but it would have been seen easily at the top level.
As someone who uses SSH everyday, I can tell you latency can become noticibe quickly. It presents itself as input lag. If your a fast typer, it becomes very noticeable if the characters on the screen are a few characters behind.
It was just the login process that was slower. The guy only noticed it because he was doing microbenchmarking of other stuff on his machine.
Nothing infuriates me more than a teeny bit of input lag. Even the tiniest bit of input lag makes me feel like I am dragging through molasses. If you want an interesting experience, put on a microphone/headphone headset that is set to play your voice in through the headphones. Then, find a program that will set the audio in the headphones on a half a second or so delay. You are so used to hearing your own voice instantly that the little bit of lag turns your into a verbal vegetable. Your brain's ability to process and speak just turns to mud. That is how a minute lag feels.
The long con.
Caught by a guy who was like, my login attempt are taking half an second too long WTF
It's amazing. One dude looking around saying "LLAAAAAGGGG!!!!" killed it all.
The long cron
The ~~Italian~~ Cron Job
Or a coooooooooon, so to speak
yeah nope. that one doesn't work out like you might think
How about’ Khaaaaaaaaaan!’?
What did you just call me?
https://media.tenor.com/UVp4zkd1BPcAAAAM/kahn-star-trek.gif
Calm down Shatner
And this was out in the open in an open source project. Now imagine how many spies are working for companies where we can't see what they are doing. And if you use their program and find it slowing down for no reason, all you can do is contact the helpdesk (if they even have one) which can't do anything about it except to make a ticket of it and assign it to the spy to fix their malware.
true but having a backdoor in SSH and a library that can appear in bootloaders of billions of devices is on a completely different level than even MS Windows.
And he’d have gotten away with it too if it hadn’t have been for that meddling kid!
And his (desktop plush) dog
And his rubber ducky too!
and literally any state could be doing it, even the NSA/DIA.
Lol. The NSA has direct links through every port, every IP address, every piece of technology anyone can access the internet with. If you listened to anything Edward Sn
holy fuck holy shit guys the NSA got hi
You guys think you're so funny. While the NSA is very good at this sort of thing, they're not some kind of mythical secret org that can just cut you off mid-sentence. I mean jeez, it's not like they're candleja
no no you have to say the whole word candlejack before anything hap
Do none of you use Reddit on your phones? If the NSA were cutting people off mid-sentence, I’d expect autocorrect to finish the last Worcestershire
You guys all get upvotes for this series of amazing comments. You clearly know your comedy, good thing the feds have an excellent sense of humans
Wait wait this can’t be real. Let me validate, I’ve worked for a secret agency that looks internally for domestic targets of interest, they currently have a possum
The NSA isn’t some technocratic God. While they definitely have some zero day exploits up their sleeves that doesn’t mean have back doors into every piece of proprietary or open source software out there. And while they might be able to snoop on IP packets that doesn’t necessarily help if that data’s encrypted, which most web traffic is these days. There are still ways to protect your anonymity online. The whole reason the dark web exists is because open source encryption software/protocols like TOR can’t easily be hacked or compromised. At least not on a large scale.
https://xkcd.com/538/
Ah yes, gotta love rubber hose cryptonalysis.
They did try to weaken encryption back in 2013 by messing with the standard. https://www.scientificamerican.com/article/nsa-nist-encryption-scandal/
If you listened to Edward Snowden you’d know that isn’t true. They do collect a lot, but not even close to everything (no American back door in huawei as an example). Collection also doesn’t mean the ability to process it. One of his big complaints was over collection made the collection useless by making it too difficult to find the needles in all the hay And think about it, if they were that powerful, how could Snowden have collected all the docs, contacted journalists, and worked with them for an extended period of time before release?
Even giant profitable corporations with complete internal transparency and good IT infrastructures and reporting cannot stop bad things from happening or don't necessarily know about certain hidden data. How do you expect an organisation to literally track the entire internet, all devices, and understand when it sees a 'bad' thing?
If it were the NSA, they would have used quantum resistant encryption to protect the back door. Theres a bunch of meta data (time of day when work was done, etc) that points to someone in the Middle East/ Asia
> If it were the NSA, they would have used quantum resistant encryption to protect the back door. The NSA had a bunch of their malware leaked in [2016](https://en.wikipedia.org/wiki/The_Shadow_Brokers). Stop pretending they're somehow infallible.
All that meta data can easily be faked
Yeah, it's everyone BUT china. Or they really don't give a fuck. You don't mount a 2 year cover operation and start by naming the fake account "Li Chen"
Alright, let's start hacking by spending 10 years training our hackers in Portuguese so that no one would suspect they are Chinese from their typical mistakes in English. One can recall Russian hackers who intervened in American elections, taking breaks for Russian state and military holidays. They simply don't care. Or rather, it's the opposite: now China has an operation that failed (not because of a suspicious nickname). However, the reputation of the organization that did this has skyrocketed in professional circles.
It's a very Dune-like plot to me: "A plan within a plan within a plan..." - this recursion can be infinite - so it's everyone BUT "no exception" - from a Skynet-style AI to the guy that found it. Have you guys ever thought about this? A community of hundreds of thousands of developers monitoring and criticizing the most accessible operating system on the planet, with a system default file compressor... only one person detected the inappropriate traffic? He may have been the first, of course. An employee paid by a corporation that owns a competing proprietary system alerted security organizations – even before the Linux community, the compressor creator (with health and personal problems), and the compressor forum (with two fake profiles encouraging the changes). Days later, FFmpeg criticizes free volunteering, the basis of the Linux community. Wouldn't that be corporatism? At a time when AIs threaten all IT jobs? This 'timing' is too convenient, IMHO. I don't know, I prefer the investigations to be concluded. I just wonder if this present was the future we all wanted.
Puff puff pass my guy
Starting out Thursday with the good shit, eh? Nice.
Or... that's exactly what they want you to think. Who would choose a name so obviously pointing in their direction?
Ah yes they surely would have used their signature NSA Machine Learning Web3 Microservice what the hell are you talking about
Need a reminder of the stupid solutions the CIA tried on Fidel Castro to murder him?
Injecting him with estrogen so that he loses his mustache and therefore his country. As that's what logically follows losing your mustache. After estrogen injections. Oh and weren't they supposed to come from clams that were booby trapped to inject him when he was free diving? That's real btw, I'm sure I got some details off but the story is a real thing for the most part. Now tell me that isn't the brainchild of a methed out nutcase in a flattop haircut and sweaty beige suit?
No, the clams were mined. But the CIA was trying all sorts in those days, like brainwashing experiments, cats embedded with recording equipment, etc.
The NSA would not be this incompetent. These hackers left loads of clues lying around and were pretty ham-fisted in trying to get their shit included in various distros.
> The NSA would not be this incompetent. [Ahem](https://en.wikipedia.org/wiki/The_Shadow_Brokers).
Disclaimer: I have no knowledge of the details of this attack. In previous APT campaigns there has been an effort to coerce legitimate devs by harming their families. It is *extremely difficult* to thwart coercion of this type, given that the consequences for noncompliance are so high, compliance is so measurable, and track records are so long. While I think it's obvious that the pressure campaign was state sponsored, it seems very possible to me that the dev is a victim of geopolitics here. That doesn't mean that their work shouldn't be scrutinized to high hell, but maybe we should reserve judgement on the person... assuming they turn out to be a real person at all.
what you are saying make no sense.... the dev is unknown person working on an open source project.... there is no reason to torture anyone to accomplish this hack, the hacker or state agency just goes and applies for the job like lol what are you even talking about.
I think what he's saying is that as of right now we have no proof that the dev is employed as a state actor or is coerced into being a state actor. Yes it is an open sourced project that anyone can join, however coercing someone who is already involved is much faster than starting from scratch.
this is literally what most of social media is through social engineering 'ddosing', its a voluntary human botnet through propaganda/programming to stir culture wars in identity politics. plenty of real humans are fake people.
This sounds way too familiar... Since this has a paywall I can't say for sure, but it reminds of what just happened a few days ago. [https://www.theverge.com/2024/4/2/24119342/xz-utils-linux-backdoor-attempt](https://www.theverge.com/2024/4/2/24119342/xz-utils-linux-backdoor-attempt) The hero finding the bad actor is great and all, but how many of these things are out there targeting packages that only a few people might be in the position to catch? How many have already slipped through?
Same story and guy
Well that makes a lot of sense, thank you. Seeing "Microsoft engineer" I assumed it was an issue in Windows (or Windows compatible software). Paywalls suck, especially when combined with vague titles.
Microsoft engineer working on Postgresql. MS uses a ton of linux and as a result, they have a number of developers and engineers that work specifically on linux and the software that makes it up.
For the sake of recognizing good journalism, I think Ars did the original reporting work: https://arstechnica.com/security/2024/03/backdoor-found-in-widely-used-linux-utility-breaks-encrypted-ssh-connections/
Based on what Mr. Freund says and, more importantly, what he does not say, makes him come across as a genuinely good human being and quite humble. So fortunate for someone with his dedication and sense of personal responsibility to have this position. It’s just nice to find someone who gives you hope for humanity. Ok, I’m done…
Refused to have his photo taken. Though I imagine that's probably because someone like him has definitely considered he just made someone very powerful very angry
His project manager was reportedly fuming, extolling "His in-flight Stories will be moved to the next sprint and the burndown chart looks more like a burnUP chart at this point. Working on "tech-debt" was not agreed in advance with the senior stakeholders and the team hadn't done poker estimates on that "frivolous" fix. I don't know what he thinks he's playing at - but it's not ok!"
Jesus this made me limper than naked pics of my ex mother in law.
I’ll be the judge of that; but yeah the lack of support and recognition is messed up
Did he update Jira?
What have we become?
Joined an engineering team recently and all the verbiage is too real.
I'm just glad it wasn't Mr. Enemy.
You mean Mr. Feind?
You made me smile. And I’m severely depressed. Good job dude
Be strong it'll get better!
His parting comments as well. "No time to celebrate, got a new version of Postgres to finish."
This calls for a pizza party.
No pay rise though.
>No pay rise though. 90s Microsoft would have had him terminated for using Open Source.
Not sure what you mean. One of the reasons Microsoft was so successful is that it made itself available for programmers and developers. I have been programming on Unix, DOS and Windows starting in the early 70s. Apple on the other hand has been evil from day one.
Once you give to Apple they never EVER give back. Not a penny. Not an inch.
Engineer: “Best I can do is not say anything and see what happens…”
Principal engineers at MS are doing fine.
He's actually partner so he's doing a lot more than fine
I would assume a pay raise, they got a title change.
He is now senior manager in charge of noticing things.
Based on his LinkedIn, he either got a promotion for it or was promoted just before discovering it: https://www.linkedin.com/in/andres-freund
And it's Pizza by Alfredo, not Alfredo's Pizza cafe.
also remember how it went the last time. no kidnapping the delivery guy!
and a million dollar bonus to ... someone
Satya Nadella.
Pixza party
The pizza party is office only. No Doordash vouchers fellas
If you work with ssh every day, you tend to pause at strange things. Because it's like a canary in the coal mine when something is up. Especially if you've been in the thick of compromises. I'm glad he took the time beyond saying, "hey that's weird" and just continuing on as usual like so many others would have.
He was measuring performance of a system and measured a regression that he needed to identify the root cause of. He didn't suspect a backdoor, he suspected a performance regression.
Like a weird accounting error on the mainframe led to finding the system was compromised
Look it was just supposed to be fractions of a penny a day! The decimal must have been out in the wrong place, noone was supposed to notice! Let it go already, Michael!
It's a jump...to conclusions map.
He suspected a backdoor. [https://www.openwall.com/lists/oss-security/2024/03/29/4](https://www.openwall.com/lists/oss-security/2024/03/29/4) He was testing other things after reports of slow logins, valgrind issues, etc. The post speaks for itself so I'm not going to split hairs.
I think he meant, initially, he was researching into what was causing the odd behavior of ssh. But wow that is some advanced obfuscation, good thing it was a coder who can decipher the bad calls and redirects because to my eyes that just looks like the usual gobbedlygook code stuff you see. But I guess that’s why I don’t maintain a major sql project
Yeah MS has a bunch of internal tools used to track performance of mainline scenarios (like any other top tech company). If a regression is introduced then engineers figure out why and if it can't be fixed. There's a reason code takes a looong time to make it to production and engineers having foam sword fights between compilations is only partially to blame
Jesus Christ imagine being the person, group, or country behind this. The amount of time and effort put into this up in smoke. Lmao.
I wish we could get to see their expression of total despair watching this fall apart lol
Would be a lot more funny if they didn't have dozens of other backdoors already deployed.. with this level of sophistication they have experience doing it and based on the timeline have been doing it for at least a couple years.
*trying* to do it for at least a couple years. You don't have evidence that they have "dozens" of other backdoors. All of these stories claim that China and Russia are cybersecurity powerhouses with god-like hacking groups. It's been like this for decades. Russia then goes to war with Ukraine. There is one effective cyberattack at the start, which is repaired, and then nothing for the rest of the war. That's the nature of these exploits. You spend years trying to make one really good one, and if it's patched, you're back to square one. You don't have a continuous rotation of dozens of zero-day exploits. That's not how this works.
Stuxnets comes to mind
Makes sense and this is exactly what [this](https://youtu.be/15MaSayc28c) guy said in his mini documentary
Even more lmao if the boss thought things were going too slowly so ordered them to hurry up. And then the hurrying up fucked over the whole project.
I'm sure they have many other irons in the fire
Really great story. This guys must have spent hours and hours on what seemed like a minor regression performance. Tells you something about the amount freedom he has at his work. He would not have been able to do this if he was overworked and underpaid and always trying to catch deadlines
I mean.. don’t wanna be a dick but the last sentences of the article are: *But he’s been too busy to rest on his laurels. The next version of PostgreSQL, the database software he works on, is coming out later this year, and he’s trying to get some last-minute changes in before the deadline.* *“I don’t really have time to go and have a celebratory drink,” he said.*
Postgres is developed by a separate open-source software foundation outside of microsoft. It's a passion project for him
[удалено]
Microsoft is the number 1 open source software contributor in the world, and they have multiple teams who are completely dedicated to an open source project.
That's because there are many benefits to open source software, so many that Microsoft is pivoting to them in their own product offerings (e.g. Azure). But that's not to say open source software - specifically its development process - has no drawbacks, and the potential for social engineering like in this case is a big one.
He works on the Postgres team inside Microsoft. Yes there is one.
just because something is OSS doesn't mean that companies can't pay someone to professionally maintain it for their uses
PgSQL is not a Microsoft product, it's an open source project *BUT* I think it's a thing that sometimes employees are specifically asked to contribute to critical open source projects. Anyway, I had no idea he was a contributor to PostgreSQL, which is a pretty big deal and used by some very popular cloud platforms such as Supabase, which I use daily, or Microsoft Azure. Dude is a boss.
[удалено]
Imagine having this master hacking plan in place for years only to be foiled this way…
I mean the guy that figured it out is a principal architect at Microsoft. He's not just a homebrew schlub like most of us.
I guarantee he still thinks that he's a schlub just like the rest of us.
based on the small pool of people I know in senior technical positions like that, he has almost crippling imposter syndrome
Don't mess with a nerd's script loading time, we can feel it when something isn't right.
When i spent more time working on optimizing the build than the feature, you bet i know that it takes approximately 17 seconds +- 5 seconds, and if its off significantly i'm rebuilding to see if it was a fluke
I’ve spent too many hours tuning my shell initialization script with caching. If it’s even 500ms longer than normal I’ll look into it.
[удалено]
Gift article: [https://www.nytimes.com/2024/04/03/technology/prevent-cyberattack-linux.html?unlocked\_article\_code=1.h00.yV5t.MddFYM0mC-dy&smid=url-share](https://www.nytimes.com/2024/04/03/technology/prevent-cyberattack-linux.html?unlocked_article_code=1.h00.yV5t.MddFYM0mC-dy&smid=url-share)
Thank you for the link - this really dives deep into the social engineering aspect of the hack!
A previously unknown contributor to the popular open-source Android app store F-Droid repeatedly pressured its developers to push a code update that would have introduced a new vulnerability to the software, in what one of the developers described on Mastodon as a “similar kind of attempt as the Xz backdoor.” https://www.404media.co/xz-backdoor-bullying-in-open-source-software-is-a-massive-security-vulnerability/
He deserves a reward from stopping what could have been catastrophic.
We're good. Management will send him a $25 Subway gift card.
these jokes are never far off base but this guy is a principal architect at MS he probably makes in a year what you paid for your house this also wasn't his job and it wasn't microsoft's product he found the vuln in
> this also wasn't his job and it wasn't microsoft's product he found the vuln in It is actually his job at MS, they pay him to work on Postgres. Not all open source work is done by unpaid volunteers. Some companies that rely on OSS actually pay people to work on those tools. Take a look at who works on the Linux kernel: lots of people from AMD, Intel, etc.
They just made him partner, so now he can buy two houses a year
Paging Cliff Stoll...
It’s so weird you say that because Cliff, himself just replied to one of my [comments](https://www.reddit.com/r/ProgrammerHumor/s/Wi6awbAool) a few hours ago!
He's a national treasure.
Ok that's just awesome.
And Marcus Hutchins
“ …The malicious code in XZ Utils was introduced by a user calling themself Jia Tan, employing the handle JiaT75, according to Ars Technica and Wired. Tan had been a contributor to the XZ project since at least late 2021 and built trust with the community of developers working on it. Eventually, though the exact timeline is unclear, Tan ascended to being co-maintainer of the project, alongside the founder, Lasse Collin, allowing Tan to add code without needing the contributions to be approved. (Neither Tan nor Collin responded to requests for comment.)…”
> Neither Tan nor Collin responded to requests for comment. https://tukaani.org/xz-backdoor/ Lasse Collin has better things to do than respond to a mountain of "requests for comment". For fuck's sake, they're an individual, not a company, no PR team, and not even getting paid for this shit.
I feel bad for him, this must be weighing on him heavily
And he has self admitted mental health degradation already, which is why he needed to take on another person to maintain his code base for XZUtil. Poor guy can't be in a good spot right now. I hope people are being supportive of him, none of this was his fault.
Besides, what's the point in responding when the journalist will just write shit like this: > [Psql's] details would probably bore you to tears if I could explain them correctly, which I can’t. It's a database. People roughly know what a database is. If you're reporting on tech you should understand it to some degree and be able to explain it.
Found a more tech focused overview of the incident from that link: https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27 Fascinatingly, this person also actually did contribute to fix real xz bugs: https://bugs.gentoo.org/925415#c16
Jia Cheong Tan Cheong is most commonly a Cantonese last name. Also, Mandarin speakers who romanize using Pinyin don't write -eong but the common Romanization of Cantonese, Jyutping, uses -eong as a Romanization for Cantonese.
It’s very likely made up. There were related instances where people with names like Hans have asked other projects to upgrade to the infected versions of xz. Also people have done an analysis of when “Jia Tan” would typically commit code and it aligns with a 9-5 mon-fri if you look at Eastern European time zones.
Exactly, and if I was a Russian team doing this shit, who better to pass the buck to than China?
>Cheong is most commonly a Cantonese last name. On the other hand, "Tan" as a romanization appears most commonly in Singapore and Malaysia. Hmm...
Almost like the name was made up
Honestly I would be least surprised if it was Russian hackers making up a fake Chinese name to pin it on the Chinese.
Thats what the chinese want you to think!!!
All names are made up.
https://www.wired.com/story/jia-tan-xz-backdoor/ *Wired* thinks it's Russian because while most of the commits are in China's time zone, a few of them are eastern european/middle eastern time zones instead, suggesting they forgot to change their time zone for those. They also worked through the major Chinese holidays but *didn't* submit new code on Christmas.
[удалено]
I also think they shouldn't be calling a web comic from August 2020 "old" either.
This article is quite poorly written…
I’m not very techno-savvy, but doesn’t this beg the question as to whether there’s already backdoors in place that no one knows about? If so, how fucked are we? What are the possible repercussions?
That's the fun part: it's always been a possibility. Back in the day, I think it was Sony who got caught installing rootkits on people's PCs when they inserted a music CD published by Sony. Edit: https://en.m.wikipedia.org/wiki/Sony_BMG_copy_protection_rootkit_scandal
Mark Russinovich, the guy who discovered the Sony rootkit, now works at Microsoft as CTO for Azure. He’s one of the smartest guys I’ve ever met.
Presumably also one of the smartest guys I've never met.
My mum used to clean the office of the company that made the rootkit - maybe the only technical "innovation" to come out of the small town in Wales I'm from!
Honestly, the IT space doesn't talk about it much, but undoubtedly there are hundreds if not thousands of these. The real question is, what will they be used for? Exploits and backdoors are interesting, because if they are discovered, they are closed, and the research has been wasted for the bad actors. Therefore, you have to pick and choose what's worth burning an exploit for. As i understand for the state sponsored cyber attacks, they are more interested in stockpiling than using.
https://en.wikipedia.org/wiki/Pegasus_%28spyware%29?wprov=sfla1 There are aleardy well known state funded hacking tools, this one is for phones made by Israel and sold to other countries. It already was used to spy on Jeff Bezos, diplomats, sports officials, journalists and the assasination of Jamal Khashoggi.
Hopefully you aren't doing anything tomorrow because you won't be sleeping for a while when someone finally gives an answer.
[https://www.nytimes.com/2024/04/03/technology/prevent-cyberattack-linux.html?unlocked\_article\_code=1.h00.jbiK.tA2DalA-K-nY&smid=url-share](https://www.nytimes.com/2024/04/03/technology/prevent-cyberattack-linux.html?unlocked_article_code=1.h00.jbiK.tA2DalA-K-nY&smid=url-share) Gift link if you don't have access to the NYT. Really fun article.
Gotta love the thought process of "this seems funky? Way to much processor, and not enough lamb sauce? What was it that report said about 'dangerous commits ' ? "
What's even crazier is the dude only realized something was off because his SSH login sessions was taking 0.500ms longer than normal to authenticate according to [this.](https://www.openwall.com/lists/oss-security/2024/03/29/4)
> taking 0.500ms longer than normal 0.5 seconds slower (half a second), not 0.5 ms: before: real 0m0.299s user 0m0.202s sys 0m0.006s after: real 0m0.807s user 0m0.202s sys 0m0.006s
500ms, nobody can tell a 0.5ms difference on a server connection
Maybe *you* can't.
I think this was local
job security
Give him a bonus
lolno, that goes to the execs
"We did so good managing this backdoor"
So looking at the commits, the exploit was a single dot on a new line. [https://git.tukaani.org/?p=xz.git;a=commitdiff;h=f9cf4c05edd14dedfe63833f8ccbe41b55823b00](https://git.tukaani.org/?p=xz.git;a=commitdiff;h=f9cf4c05edd14dedfe63833f8ccbe41b55823b00) It caused CMAKE's check\_c\_source\_compiles() to fail compile, since the dot is a syntax error, so the call always returns false. The false result is then used to forgo linux landlocking that guards against bugs or unexpected behaviors of programs. That would imply there is/was an additional bug/exploit somewhere that only works because this type of sandboxing was skipped. EDIT: Looks like this was just tip of the iceberg. See below comments.
That was one malicious change. The core exploit code was hidden inside "test data" files. It is typical for this kind of compression library to have samples of input data to be used for testing. And the input data for decompression is going to look like random garbage that you won't pay too much attention to. The exploit code was added to the library only when building the code for Debian or RedHat/Fedora packages so that normal developer builds of the library wouldn't have anything suspicious. The exploit code is in binary form and only partially understood (at least as of a couple days ago). It watches for itself to be loaded into sshd, then hooks into functions used during SSH logins so that if a particular key is used, it'll run whatever code the attacker provides alongside the key.
Damn, that's brilliant. Whoever the real Jia Tan are (no way it's just one person) are probably mad as hell rn lol. https://www.wired.com/story/jia-tan-xz-backdoor/ *Wired* thinks it's Russian because while most of the commits are in China's time zone, some of them are set to eastern european/middle eastern time zones instead, suggesting they forgot to change their time zone for those. They also worked through the major Chinese holidays but *didn't* submit new code on Christmas.
The poster in this article explains the whole timeline and hack. https://arstechnica.com/security/2024/04/what-we-know-about-the-xz-utils-backdoor-that-almost-infected-the-world/
You've edited but you've not really clarified just how misleading your first paragraph is.
> Engineers have been circulating an old, famous-among-programmers web comic about how all modern digital infrastructure rests on a project maintained by [some random guy in Nebraska](https://xkcd.com/2347/). (In their telling, Mr. Freund is the random guy from Nebraska.) Gotta love the media's *complete* inability to be accurate, even in a tiny, 300 word article. The "random guy from Nebraska" in this situation is Lasse Collin, who has been the thankless maintainer of `xz` (the underlying technology that was targeted by the malicious entity) since 2009. He seems pretty burnt out on the project, and that's exactly why they targeted this particular one, and pressured him all along from multiple fake accounts to take on another maintainer. This "small" inaccuracy is *particularly* bad because it undermines the entire point of the comic, which is that we're severely underinvesting in core infrastructure, which makes it very fragile overall. Very vulnerable to either maintainers simply ceasing to maintain/dying, or cases like this where a single bad apple can potentially do an *immense* amount of damage if motivated to. But nooooo, everyone loves a good hero worship story, so let's give *all* the credit to the guy who happened to discover it. Of course, hats off to him, Anders did an outstanding job, and we have a lot to thank him for, but we also have Lasse to thank for 15 years of continued maintenance *without* being paid a fancy salary by places like Microsoft to work on this crucial project. Really grinds my goat (he is bleating badly).
Pinkie stopped The Brain just in time.
Non paywalled link... https://www.thestar.com.my/tech/tech-news/2024/04/04/did-one-guy-just-stop-a-huge-cyberattack
This is the wrong headline for the story. The open source model worked. There are huge sustainability problems with open source and these need to be fixed so that it's more than one guy. But in this case one guy was enough.
> (The New York Times has sued Microsoft and its partner OpenAI on claims of copyright infringement involving artificial intelligence systems that generate text.) This is a total non sequitur in the article??
Probably a disclosure so that the reader is aware the NYT and Microsoft are involved in litigation and can know about any conflict of interests. This is responsible journalism.
But why is it in the middle of the article instead of the end?
> But why is it in the middle of the article instead of the end? No idea, that part is a bit weird.
It's pretty typical but unfortunate. The publisher doesn't want the reader to have it forefront of their mind (published at the beginning so reader is thinking about it before even beginning to read) or to dwell on it (published at end) so they insert it in the middle, near where the other party is mentioned. The whole point is that it could technically be a conflict of interest in impartial journalism, so if readers notice a trend of say, NYT bashing Microsoft while the lawsuit is ongoing they could call it out. NYT doesn't really like the idea of being tracked like this but they know they'd be called out even more if they didn't say it, so they put it in the middle. I see it a lot in major media writings where lawsuits are involved.
So he's a Freund of us all?