Hidde

our first AB position paper is out: https://www.w3.org/TR/llms-standards/

we tried to very briefly discuss LLMs in the standards process: where they could work and where they could be a problem, including guardrails to do it “responsibly”.

finding common ground in the AB is part of what we do and I'm glad our first attempt was published today!

Use of Large Language Models in Standards Work

As Large Language Models (LLMs) become increasingly synonymous with “AI”, and are used by people within our community, we want to highlight considerations around different ways in which LLMs can be useful or problematic when it comes to leveraging them in standards work at W3C.

Mar 24 at 8:00pmWeb
Show threadHidde4d ago
(we thought we'd start with something uncontroversial :-) )
Show threadAdrian4d ago
@hdv lol
Show threadDave Rupert4d ago

@hdv I think the "subtle incorrectness" note is a really good callout. It can take a lot of maintainer/reviewer effort to QA/dispell/validate those kinds of mistakes.

I also liked (and think about often) that callout you made in a Github thread awhile back around "asymmetry in thinking"... like if an expert and spends time building a thoughtful reply or presenting months of research only to be met with "well claude said..." that's really damaging to thoughful and informed discourse.

Show threadHidde4d ago
@davatron5000 thank you! these are def some of my biggest concerns with all this.
Show threadTim Severien4d ago

@hdv This is nice and all, but where is the merch? I need this on a t-shirt to thwart NPC posing as colleagues!

Jokes aside, this is good stuff! I am curious: how come the benefits section structured into subsections, and the risk section is not?

Show threadAdrian4d ago
@hdv A colleague recently suggested that we could translate a German spec of a metadata profile to English with the help of LLMs in an afternoon. It took years, many meetings and involved several implementors to bring it into a good state. I found it both naive and disrespectful to think this could this easily be re-written by an LLM as a solid spec in another language.
Show threadAdrian Roselli, pH03d ago

@hdv Where can we file suggested edits?

• “In addition to basic spellchecking, and editing […]” Existing deterministic algos already do this well, without emptying lakes.

• “[…] they're a great tool to use when trying to come up with human friendly names […]” This sounds like an explicit recommendation, which, feh.

• Since sustainability was considered out of scope for this issue, would like to ref for balance consideration: https://github.com/w3c/ai-accessibility/issues/15

Show threadHidde3d ago
@aardrian feel free to file in https://github.com/w3c/ab-public/ (or let me know if you're happy for me to do so on your behalf, not intending to steal your thunder)
Show threadAdrian Roselli, pH03d ago
@hdv Thanks!
Show threadAdrian Roselli, pH03d ago
@hdv Done: https://github.com/w3c/AB-public/issues/308
Edits for spellchecking, naming, sustainability · Issue #308 · w3c/AB-public

Seeing the Use of Large Language Models in Standards Work Note today for the first time. I have two suggested edits and one related issue. “In addition to basic spellchecking, and editing […]” I'm ...

GitHub
Show threadHidde3d ago
@aardrian thanks for taking the time!