Mastodawn

DrSpiderMatt Jun 4, 2024

It just clicked in my brain. What I haven't been able to articulate about why I'm so anxious about #Windows Recall. I'm sure others have already gotten to where I am.

It's worse than "a system that tracks everything you do" and stores that info in a basic database that could be easily compromised.
It's worse than a nanny surveillance tool for companies to spy on their employees.

It's inescapable.

It doesn't matter if I make a dozen "how to disable recall" tutorials. The second YOUR data shows up on someone ELSE'S screen, it's in THEIR recall database.

It won't matter if you're a master #security expert specialist. You can't account for EVERY other computer you've ever interacted with. If a family member looks up an old email with your personal data in it, your data is now at risk.

If THEIR system is compromised YOUR data is at risk.

I just went from "vague feeling of unease" to "actively writing templates to canvas elected officials, regulators, and attorneys general."

@SomeGadgetGuy remember the "timeline" feature in Windows 10? That was a bit creepy as well, this is timeline on steroids

SomeGadgetGuy Jun 4, 2024

@mountdiscovery completely unhinged and un-containable.

FoolishOwl Jun 4, 2024

@SomeGadgetGuy @mountdiscovery Microsoft depends on institutional clients, especially government agencies, many of which have stringent legal rules about access controls. There's no way Recall can be compatible with those rules.

What's astonishing me right now is that I would have expected a whole lot of Microsoft's clients to push back, "We'll have to stay with Windows 10", but I'm not seeing any hint of Microsoft backing down.

@foolishowl yup, and even though this feature is only available for newer PC with NPUs folks already hate this feature. No way it rolls out in the near future.

Microsoft should just fork Windows for consumer and professional i.e enterprises

A new OS from the ground up with all these gimmicks and Co-pilot bloatware and the other the continued legacy OS that is still compatible with all kinds of software and services.

gigantos Jun 4, 2024

@mountdiscovery @foolishowl how can it not ship?

Machines are rolling out of shops on June 18th. No way they are not already tanked with the shipping version. Hardware release is a slow process, you can't roll back on a whim.

@gigantos "roll out" as in product messaging from MS after this PR blowback

The end user is too busy with real life to care about this and how it works.

FoolishOwl Jun 4, 2024

@gigantos @mountdiscovery I get a lot of advertisements from Lenovo, and all their new laptops have NPUs.

Intel has been saying that going forward, all their processors will have NPUs. I haven't looked as closely at AMD, but I know they're advertising gaming laptops with NPUs.

I think we have to treat hardware made after 2023 as unreliable. Probably after 2022 as well.

Also the Linux Foundation and OSI have public AI projects, and Red Hat, so I expect the Linux kernel will enable NPUs.

mnemonicoverload Jun 4, 2024

@gigantos @foolishowl @mountdiscovery NPU's aren't inherently a negative. They're just a specialized co-processor for handling low precision float math quickly in parallel and with relatively lower power consumption than using a CPU or GPU for that task. From a hardware perspective it's a non issue. What software you choose (or are forced) to run that takes advantage of an NPU is where the potential problems are.

NullDott 🎩 (Comms Open 3/4)Jun 5, 2024

@mnemonicoverload @gigantos @foolishowl @mountdiscovery Indeed, NPUs aren't the problem per se, the problem is Windows forcing this "revolutionary idea" on everyone by default, making Windows machines with NPUs a high level risk, Linux would still be reliable for newer machines as long as distros don't put "AI desktops" as they're main focus.

mnemonicoverload Jun 5, 2024

@gigantos @foolishowl @NullTheFool @mountdiscovery Yeah, exactly. There's also nothing stopping Microsoft from rolling out this same "feature" to Windows PCs that don't have NPUs. It would burn a lot of CPU cycles and negatively effect performance to some extent, but there's nothing exclusive to NPUs here.

FoolishOwl Jun 5, 2024

@mnemonicoverload @gigantos @NullTheFool @mountdiscovery I don't really understand what NPUs do. I'd thought they used GPUs for generative AI processing. What's most struck me is that Intel is making such a big deal of emphasizing that they will now include NPUs on their processors, implying they're as important as GPUs for consumer hardware. So part of my worry is seeing that they're going all in on this on the hardware side.

@foolishowl yes they are...

NPU becomes a great marketing tactic and differentiating factor from other chip manufactures.

gigantos Jun 5, 2024

@foolishowl @mnemonicoverload @NullTheFool @mountdiscovery the simplified explanation is that if you want to do the basic math operations that is used by LLMs, image processing, or machine learning, the NPU will do using less power than a CPU or GPU.

mnemonicoverload Jun 5, 2024

@gigantos @foolishowl @NullTheFool @mountdiscovery There's a lot of confusion of the use case for NPUs among the general public right now because they are useful for accelerating some specific "AI" tasks but the general AI hype train is all about LLMs. The current NPUs available in production CPUs are nowhere near complex enough to be useful to run complex LLMs on local hardware, we still need data centre level grunt for that.

For a better representation of what kinds of tasks are well suited to running on NPUs we can look at what Apple is currently using their very similar NE cores on their own silicon for in iOS, like facial and object recognition for automated tagging of photos and video, handwriting recognition, image upscaling, automated audio tagging, voice recognition, background noise removal, etc. All (potentially) useful stuff that falls well outside the current AI buzzword hype in the public consciousness.

gigantos Jun 5, 2024

@mnemonicoverload @foolishowl @NullTheFool @mountdiscovery this is true, but the 10 TOPS NPU on apple M1 can run quantized llama at 30 tokens per second on a friends machine. The ones required by windows are supposed to be 4x faster. And with Microsofts research into their Phi model, it is absolutely feasible to run a reasonably capable LLM locally.

mnemonicoverload Jun 5, 2024

@gigantos @foolishowl @NullTheFool @mountdiscovery I'll take your word for it since that's well outside my wheelhouse. The point I was trying to make was more that NPUs are broadly useful for a variety of tasks that people don't typically think of when hearing the word "AI". For the general public (at least in the midst of the current hype) AI = LLM and therefore NPUs must be a bad thing that can't be trusted to exist on our computers when in reality they're just a type of coprocessor that does certain types of math more efficiently than a general purpose CPU can.

Steffen Christensen Jun 12, 2024

@mnemonicoverload @gigantos @foolishowl @NullTheFool @mountdiscovery I mean, not data center-level grunt -- you can run Llama 3-70B at home on a ThreadRipper + 2 A100 - 80 GB GPUs, which run 710 watts after the water cooling, but it'll cost you $50k for the hardware.

You could plug it into the wall socket in your kitchen.

Steffen Christensen Jun 12, 2024

@mnemonicoverload @gigantos @foolishowl @NullTheFool @mountdiscovery The rest of your argument is rock solid, however, all the rest of the AI is generally loved by the audience.

Billy Smith Jun 4, 2024

@foolishowl @SomeGadgetGuy @mountdiscovery

When chatting with colleagues from the London Tech-Financing meet-up's about this, they said that people from UK financial regulator are already investigating. :D

From the initial conversations, it sounds like no regulated industry will be able to use this, so that cuts out 80% of MS's most profitable corporate clients. :D

SomeGadgetGuy Jun 4, 2024

@BillySmith @foolishowl @mountdiscovery
SO glad to hear that! Hoping more corps and government agencies follow suit.

Billy Smith Jun 4, 2024

@SomeGadgetGuy @foolishowl @mountdiscovery

No one wants to be the person who got their employer fined by the regulator, as that's a death-knell for their career. :D

SomeGadgetGuy Jun 4, 2024

@BillySmith @foolishowl @mountdiscovery
We can only hope that this impacts a glut of middle managers...

Billy Smith Jun 5, 2024

@SomeGadgetGuy @foolishowl @mountdiscovery

Other gossip that i heard. :D

Some of the USA insurance companies are not happy. :D

They're insuring USA hospitals against HIPPA breaches, and they are looking at invalidating any claims on breaches caused by Recall... :D

The USA healthcare industry will not be able to use this software. :D

Billy Smith Jun 5, 2024

@SomeGadgetGuy @foolishowl @mountdiscovery

This will be a similar situation to the insurance companies refusing to provide mortgage insurance for buildings in Florida flood-zones. :D

Billy Smith Jun 6, 2024

@SomeGadgetGuy @foolishowl @mountdiscovery

Think how many problems patient's have when dealing with insurance companies. :D

Now add in longer contracts with way more money at stake, as well as the insurance companies having more to lose if things go wrong... :D

Even the Lloyds' Names Syndicates are looking carefully at this situation. :D

It's already affecting the Futures Markets and the Options Markets. :D

Billy Smith Jun 6, 2024

@SomeGadgetGuy @foolishowl @mountdiscovery

Like most of MS's screw-up's, this was obvious and avoidable... :D

Plus ca change... :D

Jo - pièce de résistance Jun 4, 2024

@SomeGadgetGuy @BillySmith @foolishowl @mountdiscovery
I hope there are many lawsuits filed.

Misuse Case Jun 4, 2024

@foolishowl @SomeGadgetGuy @mountdiscovery >> Microsoft depends on institutional clients, especially government agencies, many of which have stringent legal rules about access controls. There's no way Recall can be compatible with those rules.

It’s incompatible with rules in the government and most regulated industries (including healthcare). It also screws with records management.

1/2

Misuse Case Jun 4, 2024

@foolishowl @SomeGadgetGuy @mountdiscovery I suspect that as far as institutional clients are concerned, Microsoft thinks it did its duty by making it so you could turn Recall off in Group Policy Objects. That doesn’t fix the problem of, e.g., random contractors’ computers taking Recall snapshots of PII, PHI, and other sensitive data.

2/2

SomeGadgetGuy Jun 4, 2024

@MisuseCase @foolishowl @mountdiscovery
It really feels like a panic response to investor trends in AI. Something they're pushing to make their stock price nudge up, and to fight the recent Apple/Meta/Google AI initiatives, rather than an actual practical, well honed consumer facing product.
It's not yet a part of 24H2, so I feel they're announcing now and they'll be "fixing it" before it delivers at scale.

Irenes (many)Jun 4, 2024

@SomeGadgetGuy please also keep in mind that once it becomes entrenched, it will be much harder to fight. so yes - if you have energy to make a fuss, the best time is right now.

@SomeGadgetGuy As far as I understood it - it was explained to me by a Microsoft Official - the database is secured in a similar way as your face-id data or fingerprint data is secured. It's bound to your account and even the admin can't access it, because it's not a password and stored locally. That's why only devices with certain security hardware and maximum enabled security features can even activate that feature. Which doesn't mean that won't change in the future.

Advanced Persistent Teapot Jun 4, 2024

@droidboy @SomeGadgetGuy this is false, it is stored in %APPDATA% as an unencrypted sqlite database, accessible by anyone on the device with administrator rights.

H/T @GossiTheDog for details.

Ben Aveling Jun 4, 2024

@droidboy Your contact is, at best, misinformed. I would suggest not treating them as a reliable source on other matters as well.

https://cyberplace.social/@GossiTheDog/112557470517456023

Kevin Beaumont (@[email protected])

Attached: 1 image Windows Central, about the only outlet giving Recall positive coverage and having articles tweeted by Microsoft staff - have updated their take after being hands on with a device. https://www.windowscentral.com/software-apps/windows-11/microsoft-should-recall-windows-recall-security-researcher-finds-microsofts-new-ai-tool-woefully-insecure

Cyberplace

Clifton Royston Jun 4, 2024

@droidboy @SomeGadgetGuy

It's already been demonstrated that it's vulnerable. Read Kevin Beaumont's articles on it.

Microsoft is relying on people not having the knowledge or background to verify what it's telling the public.

Robin Jun 4, 2024

@droidboy @SomeGadgetGuy
There's more description of how it's actually, erm, "secured" here: https://cyberplace.social/@GossiTheDog/112557470517456023

@GossiTheDog has written quite a lot about this that's worth reading.

Kevin Beaumont (@[email protected])

Attached: 1 image Windows Central, about the only outlet giving Recall positive coverage and having articles tweeted by Microsoft staff - have updated their take after being hands on with a device. https://www.windowscentral.com/software-apps/windows-11/microsoft-should-recall-windows-recall-security-researcher-finds-microsofts-new-ai-tool-woefully-insecure

Cyberplace

Matt Hardy 3.11 for Workgroups Jun 4, 2024

@droidboy @SomeGadgetGuy I think if you're asking microsoft officials how secure it is then you're not getting both sides of the story. Mandy Rice-Davies applies. Investigations from security researchers are telling a different story. https://doublepulsar.com/recall-stealing-everything-youve-ever-typed-or-viewed-on-your-own-windows-pc-is-now-possible-da3e12e9465e There is now a proof of concept hacker tool called total recall that allows an intruder with access to your computer to dump out and exfiltrate the recall database within seconds.

Stealing everything you’ve ever typed or viewed on your own Windows PC is now possible with two lines of code — inside the Copilot+ Recall disaster.

I wrote a piece recently about Copilot+ Recall, a new Microsoft Windows 11 feature which — in the words of Microsoft CEO Satya Nadella- takes “screenshots” of your PC constantly, and makes it into an…

DoublePulsar

Asta [AMP]Jun 4, 2024

@droidboy @SomeGadgetGuy I know others have already discussed why this isn’t quite true below, but as a security rule of thumb one should always keep in mind that if the data can be accessed legitimately, it can also be accessed illegitimately. There are at least as many ways into a building as there are doors, so to speak. Hence why the safest data is the data you never collect.

Lillian Violet Jun 4, 2024

@droidboy @SomeGadgetGuy But the data is still processed into a model. And most likely that model will generate analytics that will be sent to HQ. And one thing my experience has taught me is that recreating data from models is easier than tech companies want to admit, and they only do this to skirt the law and not directly upload your data.

gunstick Jun 4, 2024

@droidboy @SomeGadgetGuy it is as secure as you, logged in user, can access it, and any program you run can access it. Which includes info stealing malware. Ohhhh

@gunstick @SomeGadgetGuy Thats the thing we need to see happening in the wild first, please. :-) I dont want to believe MS is building it like that. As what I was told was, that its in the same container as your fingerprint.

gunstick Jun 4, 2024

@droidboy @SomeGadgetGuy no. You cannot read out the fingeprint to make copies of. But you can read out recall data. So it cannot be the same container, else you would not be able to read the recall data.

@gunstick @SomeGadgetGuy Can't wait to see how they actually do it.

gunstick Jun 4, 2024

@droidboy @SomeGadgetGuy it's a simple mysql file on your disk. That's all.

@gunstick @SomeGadgetGuy I will repeat it until we see it in real life on June 16th: We don't know yet :-) You can tell me than that you already knew it. But wait until it actually gets released and somebody could look at it, please! :D

gunstick Jun 4, 2024

@droidboy @SomeGadgetGuy people already had a look at it, and it is horrifying.
He somehow managed to get the software. So here you go: https://doublepulsar.com/recall-stealing-everything-youve-ever-typed-or-viewed-on-your-own-windows-pc-is-now-possible-da3e12e9465e

Stealing everything you’ve ever typed or viewed on your own Windows PC is now possible with two lines of code — inside the Copilot+ Recall disaster.

I wrote a piece recently about Copilot+ Recall, a new Microsoft Windows 11 feature which — in the words of Microsoft CEO Satya Nadella- takes “screenshots” of your PC constantly, and makes it into an…

DoublePulsar

@gunstick @SomeGadgetGuy I read that, and I assume he got a view on a MVP. I do not want to believe this is actually the final product.

gunstick Jun 4, 2024

@droidboy @SomeGadgetGuy how do you imagine the final product will be used?
To be secure it would need a public key which encrypts everything going into the database and a secret key stored on an external device you have to activate each time you want to interact with the database. Think yubikey.
If the currently logged in user can interact with recall data the same easy way as windows search, then it is acessible to every malware you may be running.

@gunstick @SomeGadgetGuy So what we know is that only Copilot needs access to that database. Is there no way to develop the access management in that way that only the software Microsoft controlls can access it? No public API needed in any way. Granted: What Copilot will show you might be accessable to the malware as this is what the user will see at the end.

gunstick Jun 4, 2024

@droidboy @SomeGadgetGuy malware will hide as a system uodate requesting elevated privileges, and the user will simply click OK as he does with all the other dozens of popups he sees per day.

@gunstick @SomeGadgetGuy There will be no way you can grant access as an admin to any other app. Only Copilot will have access. Is that possible in windows?

gunstick Jun 4, 2024

@droidboy @SomeGadgetGuy if copilot is running in another vm under the same hypervisor as windows. Oh windows is not running as a VM?
Well then: no.
If the recall database is encrypted and is on the same storag as the rest of the OS, then copilot has the password. And having admin privileges will open that up to inspection. Game over.

SomeGadgetGuy Jun 4, 2024

@gunstick @droidboy

That's what I want to see. Before you can interact with Recall you HAVE to plug in a security key. We just know MS won't do anything like that, because it'll then look scary (which it should), and they want it to be "frictionless".

The idea of it being more secure, Microsoft is doing bad infomercial acting about how inconvenient and complicated security is.

SomeGadgetGuy Jun 4, 2024

@gunstick @droidboy yup. Stored in the user App data folder.

gunstick Jun 4, 2024

@SomeGadgetGuy @droidboy ah, yes. That vaguely hidden folder, so it must be secure™
I wonder if they now store the fingerprints also there sonthey can still claim that it's stored at the same place.

SomeGadgetGuy Jun 4, 2024

@gunstick @droidboy
Security through obscurity!

synlogic Jun 4, 2024

@SomeGadgetGuy @gunstick @droidboy my ideal: security through not totally blatant forehead-slapping levels of idiocy & self-ownage

@synlogic @SomeGadgetGuy @gunstick I am so glad you got the opportunity to show how much better you are than the security guys from Microsoft.

D S Jun 4, 2024

@droidboy shut up

@dantescanline Dude. :-) That's not the style you want to be remembered for. Do you?