Carl T. BergstromNov 15, 2022

It looks like Meta just quietly (?) rolled out a new Large Language Model trained up on the scientific literature. At first glance it's not terrible—I was thinking I had reviewed worse—but it starts talking nonsense after a while.

Still, interesting and makes me think about the research papers I'll be receiving from students in coming years.

https://galactica.org/?prompt=literature+review+on+costly+signaling+theory&max_new_tokens=1400

literature review on costly signaling theory - Galactica

Show threadCarl T. BergstromNov 15, 2022
"If the seller does this, then the buyer will buy the phone at a higher price than it is worth, and the seller will lose money."
Show threadCarl T. Bergstrom

Feeling a bit bad now TBH.

Maybe I didn't give them enough credit.

I thought it was just a language model that produced middling results. I didn't realize this was "a new interface to access and manipulate what we know about the universe."

Nov 15, 2022 at 10:12pmWeb
Show threadCarl T. BergstromNov 15, 2022

Currently having a blast playing with Amazon's lex.page, thanks to @tonic.

This is quite a nice plot twist it threw in there.

Show threadJoseph P.Nov 15, 2022
@ct_bergstrom I’ll make sure to connect you with the creator. There’s something about this GPT-3 AI that’s almost there (GPT-4 is coming soon!) , combining the lex page with something similar for spreadsheets and btw it can kind of code an algorithm and I’m guessing that s what will be in the most basic toolbox for any one basically.
Show threadJoseph P.Nov 15, 2022
@tonic @ct_bergstrom one last thing : try to write something like an undergrad essay on a “science topic” it’s pretty good at it!
Show threadMike Miller, PhDNov 16, 2022
@tonic @ct_bergstrom isn't this a low bar? Undergrads are terrible at writing essays. 😉
Show threadJoseph P.Nov 16, 2022
it's true, but some people even need to teach them , so it could be quite a nice tool - you can probably design a full hour long presentation at undergrad level in 30 minutes or so with the thing... maybe...
@mlmillerphd @ct_bergstrom
Show threadstephenduffNov 15, 2022
@ct_bergstrom sounds like you've found the new FTX 😂. Thanks for the server recommendation
Show threadMike Taylor 🦕Nov 15, 2022
@ct_bergstrom Galactica has a ways to go yet. The answer is was looking for was "CM 84, CM 94, CM 307 and casts from other specimens", but I would have accepted "CM 84".
Show threadMike Taylor 🦕Nov 15, 2022
@ct_bergstrom Much better answer on this one, "which dinosaurs are included in the group diplodocidae". It missed Torneria, Kaateodocus, Galeamopus, but the ones it did name are correct. Sort of "good enough for high-school homework" level.
Show threadMike Taylor 🦕Nov 15, 2022
@ct_bergstrom Oh dear ... And it seems so certain! But, no Xenoposeidon does NOT belong to Tyrannosauridae!
Show threadMike Taylor 🦕Nov 15, 2022
@ct_bergstrom It's pretty darned poor on the causes of Brexit, too.
Show threadMike Taylor 🦕Nov 15, 2022
@ct_bergstrom LOL at the 1977 publication date of Marsh 1880.
Show threadXavier F. C. Sánchez DíazNov 15, 2022
@ct_bergstrom I've given up at this point with LeCun's visions on AI, to be honest. And I believe this is just another nail in the coffin.
Show threadCoprolite9000Nov 15, 2022

@ct_bergstrom
Definitely accessing and manipulating what we know about the universe here...

(Didn't even specify units! Hopeless.)

Show threadAlbert CardonaNov 15, 2022
@coprolite9000 @ct_bergstrom How does it fare with the Buffalo buffalo ... sequence?
Show threadCoprolite9000Nov 15, 2022
@albertcardona @ct_bergstrom
Underwhelmingly!
Show threadLambdaDuckNov 15, 2022

@ct_bergstrom I wonder how feasible it would be to teach these language models to cite sources?

many things are cited almost verbatim from an original source, so it would be very useful if it could tell the source in those cases

Show threadLambdaDuckNov 15, 2022

just putting an identifier of each source in the training data would probably not give reliable enough results, since it may decide to just invent its own sources. on the other hand, assigning a bunch of sources to each weight would multiply the size and cost by a lot

maybe there would be some compromise with relatively low cost that still provides reasonable accuracy on which were the most important sources for an answer?

Show threadMichael BurkeNov 15, 2022
@ct_bergstrom The intro got stuck in a loop for me. Fun failure mode.
Show threadIain BancarzNov 15, 2022

@ct_bergstrom

"a new interface to access and manipulate what we know about the universe."

I guess technically it is, but so are a lot of things. The phone number of the new pizza delivery place down the road, for instance. 😀

Show threadAdvadnounNov 15, 2022
@ct_bergstrom Meta AI is so obnoxious. They really ought to be putting disclaimers about bias and misinfo there instead of hype shit.
Show threadRaef KaziNov 15, 2022
@ct_bergstrom i wonder if researchers at Meta use this tool for their own research, and if so how. Also thought it was a neat trick to showcase the tech with the usual LLM pitfalls (has some interesting results), but this kind of messaging just sends the wrong information across, especially to non experts seeing it for the first time
Show threadQuantum MechanicNov 16, 2022
@ct_bergstrom What the frak?