@jwz it's weird, I'm very firmly in the "cryptocurrencies are a waste of everyone's time" camp, but the more time I spend with large language models the more convinced I am that they are going to let me solve all kinds of problems that I couldn't solve before

Nailing down exactly what those problems are is a lot more involved than I think most people expect though

@simon What kind of problems?

Even if there are upsides, the downsides seem pretty severe. These systems are optimized for bullshit and lies, like, that's their core competency to which all use degrades. See also "facial recognition is the plutonium of AI". https://jwz.org/b/yjMP

@amyhoy @jwz I'm talking about prompts like "Here is a copy and pasted police report. Return JSON with the names of the mentioned officers and the date of the incident"

My hunch here is that spot checks on the results could help tell if it's working well enough, and that the end results would reach the same level of accuracy as asking human data entry people (who are also infallible) to do the same task

If there's a better way to do this than using a language model I'm interested to hear it

@simon @amyhoy
But how could you ever trust that data? You are asking for *facts* but the system is optimized to produce *believable answers* which are not at all the same thing.

Suppose the system optimizes its march to the goal by just making up some numbers that subtly (or not so subtly) tilt the data one way or another. Now you've built a black box to confirm your biases.

And the black box, by its nature, cannot "show its work" without lying.

@jwz @amyhoy The black box thing is why I'm finding this whole space so utterly beguiling

I hate that it's a black box. But I've spent my entire career working with computers that do exactly what you tell them... and now I'm faced with one that very much does not do that

It's like someone's given me a spell that raises actual dragons from another dimension and challenged me to try and tame them!

@jwz
@simon @amyhoy

Actually, we generally don't do a good job of regulating things that hurt working class people.

@amyhoy @ian @sayrer @jwz It's a next-token-predicting language model, so using it for math is very much the wrong application of it - that's one of the many reasons I keep trying to convince people that these things are deceptively difficult to use effectively

The idea that a computer can be bad at math is very counter-intuitive!

@anildash @simon @amyhoy @ian @sayrer @jwz I just wish it were easier to separate the wheat from the chaff.

As someone who is generally positive toward this technology, I can still benefit from thoughtful critiques on its efficacy informed by experts who understand its limits.

Even amongst those who see value here, it’s important to understand what we’re dealing with and how far it can be taken safely.

@jeff @anildash @simon @amyhoy @ian @sayrer @jwz But much of the discourse seems like motivated reasoning, as it’s perceived as posing a risk to certain professions.

For many it seems less about the technology’s limits and more that they don’t _want_ it to be/get good.