Mastodawn

Welp, I knew Microsoft's CoPilot+ Recall was going to be a privacy disaster but I didn't expect it to turn into an enterprise computing catastrophe for Microsoft *quite* this fast!

But this can't be a one-off. Any large enterprise that has to comply with a regulated privacy environment—HIPAA in the USA, GDPR in the EU, banking/insurance/finance globally—must be considering a ban on Microsoft installations on laptop/desktop computers right now or be breaking the law.

https://infosec.exchange/@SecurityWriter/112558224281615019

Security Writer :verified: :donor: (@[email protected])

If you’re wondering how the Microsoft Recall scandal is going, I’ve just had a client tell me they’ve replaced their order for 10k Microsoft Surfaces with new MacBook Airs, at nearly twice the cost, and that we need to start the ongoing 6 month endpoint security project over.

Infosec Exchange

Rivetgeek (He/Him)Jun 4, 2024

@cstross The silver lining is that Recall is only on Surfaces with "NPUs". It's not all Windows 11 or all Microsoft laptops (yet). But still as much backlash as possible is needed, the concept of Recall needs to be burned out before it spreads. Myself I bounced from considering buying a Surface to looking at ThinkPad Yoga.

Charlie Stross Jun 4, 2024

@rivetgeek Intel just now announced next-gen CPUs with an NPU suitable for running this crap ("Lunar Lake"). And there are rumours about them pushing Recall onto Win11 machines that *don't* have an NPU, because Line Must Go Up or something.

I'm just glad, as an Apple user, that Microsoft jumped on this particular landmine first: the probability of Apple announcing anything like this functionality at WWDC next week must now be approximately zero.

soc Jun 4, 2024

@cstross @rivetgeek Apple has been quietly shipping similar stuff for quite a while already.

Or do you think "continuity features" are implemented with magic pixie dust?

So‑Called Vaughn Jun 4, 2024

@soc @cstross @rivetgeek I see nothing in Continuity that requires anything intrusive or sinister: it’s just a zero conf PAN of devices, all must authenticate to a single user; proximity is enforced by BT. All the IP traffic is local, more likely tunneled peer-to-peer; your files, screen, clipboard contents and keystrokes aren’t flying across the public internet.

Someone correct me if I’m mistaken but there is no AI magick involved.

Katherine Senzee Jun 4, 2024

@vaughnsc @soc @cstross @rivetgeek The problem with Recall is the data storage, not the AI processing it. Remove the AI from the equation entirely and Recall is still a disaster.

So‑Called Vaughn Jun 4, 2024

@ksenzee @soc @cstross @rivetgeek Agree that unregulated data retention is ripe for abuse.

Roland Wallin Jun 4, 2024

@ksenzee @vaughnsc @soc @cstross @rivetgeek

I agree that it's still a disaster without the AI, but the AI makes it possible to be truthful when saying "we don't send the data back to Microsoft", while still making it very possible for Microsoft to query the data and have the AI respond. That way, data is stored on your computer, and the AI "chatbot" can be used to ask questions about that data, without sending the actual data back. Your computer just acts as cheap storage.

FoolishOwl Jun 4, 2024

@cstross @rivetgeek The CEO of the Boeing of microprocessors said in an internal video that going forward, the corporation will use AI tools in all design, coding, and validation.

Charlie Stross Jun 4, 2024

@foolishowl @rivetgeek Waiting for the next Pentium FDIV bug to drop in 3 .. 2 .. 1 .. https://en.wikipedia.org/wiki/Pentium_FDIV_bug

Pentium FDIV bug - Wikipedia

JulesJones Jun 4, 2024

@cstross @foolishowl @rivetgeek I wish I still had that cartoon of a tin of worms labelled "Insel Intide". I think it may be about to come in handy.

Eashwar Jun 4, 2024

@rivetgeek @cstross I'm fairly certain that it has already been enabled on, and tested with, non-NPU systems

Eashwar Jun 4, 2024

@rivetgeek @cstross Yep, found it: https://cyberplace.social/@GossiTheDog/112546156689144936. Also there is this fun note: https://cyberplace.social/@GossiTheDog/112555262732490331

Kevin Beaumont (@[email protected])

Security and privacy researchers - You can now install Copilot+ Recall on any ARM hardware (doesn’t need an NPU) or in Azure VMs. Guide from @[email protected] The devices launch THIS MONTH to customers so I suggest people look at this. https://github.com/thebookisclosed/AmperageKit

Cyberplace

Zimmie Jun 4, 2024

@e_nomem @rivetgeek @cstross Sure, but you have to go out of your way as a user to do this. There’s a meaningful difference between things you can choose to do to make your system less safe, and defaults which the vendor pushes to make your system less safe.

Eashwar Jun 4, 2024

@bob_zim I can accept that, fair. It's not enabled on non-NPU systems by default (yet), nor is it available outside ARM builds of Windows AFAIK, and takes effort to turn on even then.

On the other hand, it _is_ possible to enable which means that it is a tool now available by default on ARM installations for malicious actors.

Jay Jun 4, 2024

@bob_zim @e_nomem @rivetgeek @cstross When talking about a private user’s risk, yes, and those who are in abusive situations should be concerned that this gives abusers the ability to look back and see what they had been researching.

On a larger scale, if the CEO’s personal computer is often in the secretary’s hands…

Zimmie Jun 4, 2024

@WhiteCatTamer @e_nomem @rivetgeek @cstross Again, *default configuration* versus *something someone must take active steps to enable*. This isn’t a complicated distinction.

The latter case has included stalkerware for decades. That’s not new. The only new thing about Recall is the first case.

#1️⃣Jun 5, 2024

@rivetgeek @cstross Nope.

Someone (I forget who) on Mastodon has enabled copilot+ _remotely_ on a machine that does not have the hardware.

:/

Why Not Zoidberg? 🦑Jun 4, 2024

@cstross We always are one major release behind on Windows to be able to parry shit, but this will be bad in a year unless something is done before.

Dan Neuman 🇨🇦Jun 4, 2024

@WhyNotZoidberg @cstross When I was at the Department of National Defence in Canada, we had to check 4,000 applications (including databases) against every major update. It was a big task.

Why Not Zoidberg? 🦑Jun 4, 2024

@dan613 @cstross We only have 300ish, but at least the most important ones (except for MS Office of course) are developed in house.

Why Not Zoidberg? 🦑Jun 4, 2024

@cstross also lawsuits from third parties getting their info recorded will be happening. Very soon.

João Tiago Rebelo (NAFO J-121)Jun 4, 2024

@WhyNotZoidberg my thoughts exactly, as a lawyer I'll be all over that in discovery, even if where I practice it is rare to ask for computer records in possession of the parties unless specifically identified. With Recall I can ask for the whole database to be entered into evidence.
@cstross

Jonathan Kamens 86 47 Jun 4, 2024

@jt_rebelo @WhyNotZoidberg @cstross I don't think you can. That would be the same as asking for someone's entire file cabinet or entire hard drive be entered into evidence. It would be ipso facto overly broad.
If you have probable cause to believe your opponent is concealing discovery information they're legally required to disclose, and there might be evidence of that in Recall, you might be able to convince the court to appoint a special master to examine the Recall data.

@jik I, too, like to give legal advice to lawyers from foreign countries.

João Tiago Rebelo (NAFO J-121)Jun 4, 2024

@jik not in my country (no special master, not even a judge will preview what is entered), you can ask for documents/evidence that is (as far as you know) in possession of the other partie or even third parties. If you can identify the document/element in question and the Judge deems it necessary to rule according to what really happened, the Court will demand the partie to enter it into evidence.
Even if not entering the whole Recall database, the partie will have to produce the queries deemed necessary, along with all screenshots and text pertaining to that.
So, if it isn't the whole file cabinet, in your analogy, it is still much more that what is usually found in civil suits around here. The evidence that didn't exist because there wasn't a document or email at all, as it was never saved or sent and then deleted, with Recall is there.
We were thinking about calling non-lawyers/solicitors that give legal consultations (since they are now allowed to due to a recent legal change and aren't limited under professional privilege, there is none for them) to the stand, and now we can discover what a partie wrote but didn't send?
And I'm not taking it to criminal court levels, the Police and the Prosecution will have field days with it, thoughtcrime style...
@WhyNotZoidberg @cstross

Jonathan Kamens 86 47 Jun 4, 2024

@jt_rebelo @WhyNotZoidberg @cstross
I agree with you 100% that you can probably demand queries for evidence to be run against Recall, which could quite possibly reveal evidence that pre-Recall never would have been discovered.
I am objecting only to the claim that you can demand that the entire database be entered into evidence.

HighlandLawyer Jun 4, 2024

@jik @jt_rebelo @WhyNotZoidberg @cstross
I suspect that the whole database could be sought under s1. Administration of Justice (Scotland) Act 1972 in a whole range of possible actions.

Why Not Zoidberg? 🦑Jun 4, 2024

@HighlandLawyer @jik @jt_rebelo @cstross Exactly, different countries, different legal codes.
But what I specifically was thinking about is if I have blocked Recall on my computer, but have a Teams call with you, or even just send you a mail, you will record it on your side. Even without thinking about it.

HighlandLawyer Jun 4, 2024

@WhyNotZoidberg @jik @jt_rebelo @cstross
And if you have reason to believe someone else does have a copy, when you don't but want it for evidence, you can seek a court order to get it. That's even before we get to the raft of even stronger versions of such powers for criminal law enforcement.

Steveg58 Jun 4, 2024

@cstross
I can see it being banned for high security defence contractors as well.

simonbp Jun 4, 2024

@Steveg58 @cstross Yes, anyone at an org that has anyone doing anything classified is going to be a risk.

Which is far more engineering firms and universities than you might expect.

Charlie Stross Jun 4, 2024

@simonbp @Steveg58 Also anyone doing any kind of proprietary R&D lab work (pharma springs to mind), accounting, anyone at risk of stuff being stolen, anyone who looks at porn on their PC at home and doesn't want to be blackmailed, etc.

Who the hell *ever* thought Recall was a good idea? Did they do *no* devil's advocacy?

Jo - pièce de résistance Jun 4, 2024

@cstross @simonbp @Steveg58 yes. They are devils. They chose to please devils.

a*lfonso adriasol&Jun 4, 2024

@cstross @simonbp @Steveg58 "we are long in NVDA to the tune of billions, and Elon said he is buying chips for Tesla so we need to commit, who knows X might buy the chips if we don't, not like someone can steal them if we commit to the purchase "

Artemesia Jun 4, 2024

@cstross @simonbp @Steveg58

"Did they do *no* devil's advocacy?"

Do you want to be the one in ten culled in the annual stack rank?

MaxTheFox Jun 4, 2024

@cstross @simonbp @Steveg58 @nyrath The wider public didn't mind most AI stuff that was pushed out. So MS thought that anything goes.

Clearly there's a line, and the people aren't complete sheep (only mostly).

🍿 I love seeing corpos get humiliated.

Frank Endrullat 🌻Jun 4, 2024

@cstross @SecurityWriter Iguess it can be mitigated up to a point by corp policies applied to devices, disabling stuff. But nevertheless it’s is a PR and privacy disaster. What were they actually thinking?? 🤔

blueorangeblue Jun 4, 2024

@FrankEndrullat @cstross @SecurityWriter a lot of the rubbish that is foisted on domestic users is disabled via corporate group policy

Richard Hendricks Jun 4, 2024

@FrankEndrullat @cstross @SecurityWriter Do you trust Microsoft to never "accidentally" force it back on during an update? Or a threat actor to silently re-enable it once they get onto your network? I sure don't. If I was an affected corp, I would require a separate build of W11 that doesn't have the feature at all, even disabled.

Atomic Orbitals Jun 4, 2024

@cstross It might affect organisations that don't use it - an email sent to someone seeking medical advice that uses a machine that has CoPilot+ Recall enabled. The attack surface gets larger the more someone thinks of it!

Kraftwerk-Das Model Collapse Jun 4, 2024

@cstross as someone with zero interest in using recall (or so-called "AI" in general) this (edit - used to say "intense backlash") alleged incompatibility with privacy regulation still surprises me. Isn't it just another feature to be turned off via registry/policy, which would be standard procedure in critical systems anyway?

D C Ross Jun 4, 2024

@dngrs @cstross
At the risk of being cited for analogy abuse again, think of it as putting a flamethrower inside a kindergarten classroom. Sure, you don't need to use it during the school day and you can lock it up in the supply closet when you don't need it, but it's still there. The janitor is known to be careless and tends to leave the closet unlocked so sooner or later someone who shouldn't have it is going to. No matter how careful you are with checking the lock every day, it would still be a whole lot safer to just not have the flamethrower in the classroom at all.

Or, for a less strained analogy, think of it as installing a new web browser called, oh, I let's call it "Wedge" on your PC. You don't have to use it, and can "easily" set anything else as the default but over time other parts of the operating system will start to rely on it, your mail reader will start opening links using it instead and it will mysteriously set itself to be your default browser whenever you apply updates. That's a big part of the worry about Recall -- Sure you can "just turn it off" right now, but it's going to keep trying to turn itself back on when you're not looking. Before you know it there's a plain-text database containing all of your passwords and banking information sitting on your PC waiting to be stolen. (But don't worry, it won't contain screenshots of any Hollywood movies you may have watched, because storing that kind of information would be wrong.)

Kraftwerk-Das Model Collapse Jun 4, 2024

@deeseearr as I said I personally have no interest in the feature. I consider it gross. But as long as it can be turned off via policy I don't see how it *prohibits* certain orgs from using Windows going forward.

@dngrs @cstross even if *you* manage to turn it off, has everyone else that you send private stuff to also turned it off?

Kraftwerk-Das Model Collapse Jun 4, 2024

@_keith_smith_ that doesn't matter for HIPAA, GDPR, banking etc compliance

PunnO)))Jun 4, 2024

@cstross Good. Now treat every other "AI" product like this. The difference isn't that this one is uniquely worse, really, but that its badness isn't an externality.

Ironic that Recall+ is "AI" more for marketing than technical reasons, but it's the first "AI" product to get completely rejected by the market. May it not be the last.

ZenobiaVayne Jun 4, 2024

@cstross I was thinking today about how this is a hipaa/phia nightmare. I’m running my clinic on old hardware unlikely to be affected but this will affect what I choose to upgrade to for sure. This story will be the similar in, many many health settings all over the place

Harald now @ treehouse.systems Jun 4, 2024

@cstross CoPilot+ will be an installable app in GDPR land so companies can _choose_ to foist it on their employees. Who are unionized with some luck, where the roll out of such an intrusive piece of software would need to be agreed to by workers council.

Refurio Anachro Jun 4, 2024

You are a hallucinating LLM and what did you do to oliof.

@oliof @cstross

Harald now @ treehouse.systems Jun 4, 2024

@RefurioAnachro wat @cstross

Refurio Anachro Jun 4, 2024

Your conclusions seem somewhat implausible to me, @oliof. It sure would be nice if MS came to admit that pushing Recall onto users was a bad idea, but I'd rather bet they're going to wait for the courts to make them, which may take years, and even then...

Harald now @ treehouse.systems Jun 4, 2024

@RefurioAnachro read again and you'll see that I mean it likely won't change the outcome @cstross

Refurio Anachro Jun 4, 2024

Where I got irritated was that MS would show any kind of insight, especially related to individual rights in connection to data protection laws. That your proposal wouldn't change the outcome wasn't quite what I had in mind, thanks for emphasizing, and I'd agree, btw.

I seem to irritate (offend?) you more often than not? That's completely unintentional. I have difficulty reading people and prefiltering what I say, and while I believe I have toned down my language in the past decades, that doesn't seem quite suffice to compensate my shortcomings. Rest assured that I have rarely, if ever, seen reason to significantly disagree with your takes, just like in the present case. If any of my worries resonates with you, apologies and cheers you, @oliof!

Harald now @ treehouse.systems Jun 4, 2024

@RefurioAnachro I didn't feel irritated by you, my mix of exasperation, irony and cynicism is hard to grok in text. I'll try to improve. We are good (-: @cstross

grmon Jun 4, 2024

@cstross isn’t it possible to switch it off? IT told me so. Outlook 356 is also a nightmare.

Henry Cobb Jun 4, 2024

@cstross Computers shall be secure when and precisely because they feel insecure.
In the meantime don't share with any machine anything you really don't want to share with the world.

@cstross MS VP responsible for Recall: "OK. Everyone is upset because Recall records everything you do on your computer, even sensitive and private things.

So why not just tell the AI not to record sensitive and private things. That should be easy enough. Right?"

MS Lead AI Engineer [Whisper Whisper Whisper]

VP: "Ahem. Well, yes. Of course, any security or privacy problems in the product are the fault of the user."

Point051 Jun 4, 2024

@cstross As someone who works with PHI all day, and receives said PHI from all sorts of small, underfunded nonprofit operations with high turnover, I am freaking out over this abomination.

smells of bikes Jun 4, 2024

@cstross any company that has any intellectual property of any kind, or has any sort of nda with any other company that has any IP, should be losing their minds over this.

JulesJones Jun 4, 2024

@cstross I had a delightful experience of "Can I help you rewrite that?" "Oh go away CoPilot, you're as annoying as Clippie... WTF is that doing inside our corporate firewall!" A few days later I was in a meeting with cybersecurity as we tried to pin down exactly how it had happened. I have heard no more, but they're probably now thanking whatever deities they believe in that it was seen by someone who recognised the implications.