If it were up to me, Apple would create the world’s first “Do Not Train” registry. Enter your domain or web page, get a TXT record or meta tag to prove ownership, prevent Apple from using your content to train any LLM.
Only a small percentage of sites would do it, I think, so the training impact would be low, but it’d be extremely meaningful for those people. And it’d be a valuable PR tool, brand-booster, and competitor-shamer. (Bonus: make it open and encourage competitors to follow it.)
@cabel (a) why would this be different than then saying they’d honor a robots.txt?
(b) this sounds like a do not track thing, but without even the attempts at legislation around it (which themselves failed, and using the header at all is pointless)
@cabel As somebody who runs a service that does at lot of user-driven web crawling for archival purposes, I'm familiar with exactly how loose it all is...
I just don't think an opt-in list is the way to create this accountability. I would prefer it be settled with case-law that says "if you train on copyright protected work you'll get mega-sued" but large companies are trying to get ahead of that by making cash deals with anyone who can afford lawyers right now. :(
@cabel @hank right, this would entirely benefit Apple’s brand and require extra work for one specific vendor.
Frankly if they did this and opened it up to third party AI providers, I would be extremely tempted to sign up just to wantonly violate it and drag down the reputation it might carry.
Yes, I’m that petty, but maybe the only computer company I’ve ever used (save one gaming PC) in nearly four decades should stop acting in ways that have me looking for the exits.
@cabel surely every single enterprise and corporation would immediately register in it given the current temperature? They would, collectively, remove most of the internet from the training model. Every lawyer & biz dev would be making it happen overnight.
This feels like something that has to happen *after* (a) contracts for licensing ‘big’ content have been signed (b) business has collectively accepted that actually training LLMs on their dross isn’t actually impacting their business.
@cabel They published the datasets used “Our pre-training dataset contains RefinedWeb, deduplicated PILE, a subset of RedPajama, and a subset of Dolma v1.6, totaling approximately 1.8 trillion tokens." here https://huggingface.co/apple/OpenELM It would need to be more of a supplier contract requirement.
Supposidly you can search the datasets as well https://huggingface.co/datasets/tiiuae/falcon-refinedweb
But yes I like the registry idea as well. With all the licensing going on, sites could provide details on licensing as well.
@cabel
That's revisionist history. Blockchain used to be everywhere. Nobody wants to admit they were building around it. Also, Microsoft did bet on blockchain, they just aimed at businesses more than consumers, offering support for it in Azure. I recall that Apple talked about it too.
Right now, we are somewhere around the peak of the AI bubble. LLM innovation has slowed and venture capital money won't last forever. I expect enshittification to arrive soon for AI and for the bubble to pop.
@cabel at least second; ServiceNow and HuggingFace have been doing it for at least a year: https://github.com/bigcode-project/opt-out-v2
https://spawning.io have also been advocating for an EU-based centralized opt-out registry (since EU law requires opt-outs starting in less than a year, but doesn't require a central registry) but not sure how far along their proof-of-concept is.
Cabel Sasser
Eric Eggert
Jason Petersen (he)
Henry Wilkinson
💀 𝓕airchild 💀
Daniel Jalkut
Dominic Hopton
gregandcin 
Blake Garner
Jari Komppa 🇫🇮
phi1997
Mayo
Drew Costen
Nicolas Ward 
jramskov
IAG
Oliver Hunt
Luis Villa
Daniel M Karlssonrobots.txt not cutting it tho. I would prefer opting out in that file to be enough.
Brett Walker