Jason MorrisI think this #FutureLaw 2023 panel on GPT4 is good in terms of a balanced view of the risks and opportunities of using tools like GPT4. What surprises me is that no one is talking about how soon GPT4 will be obsolete, and replaced by something that improves over the previous iteration just as significantly. #LegalTech #LawFedi
Lisa Fast@lexpedite Exponential!
L Thorne McCarty@lexpedite
Some evidence that GPTs may have reached a plateau?
https://www.theverge.com/2023/4/14/23683084/openai-gpt-5-rumors-training-sam-altman
Some evidence that GPTs may have reached a plateau?
https://www.theverge.com/2023/4/14/23683084/openai-gpt-5-rumors-training-sam-altman
@ltmccarty that article just says you can't assume that new versions are better than earlier versions in any stable degree. Granted. But no one is assuming. It is objectively getting significantly better. That they haven't started "training" GPT5 yet doesn't mean anything. I don't see any evidence of a plateau, and I see lots of evidence to the contrary.
Damien Riehl@lexpedite @ltmccarty — Two ways this could play out:
1. OpenAI feels threatened by the many free + open-source competitors (e.g., Eleuther, Dolly-2), and wonders whether the time/expense of generating a Foundational Model is worth it — when they're competing with "free."
2. OpenAI — with Microsoft money — takes a run at improving the existing GPT-4 model incrementally. Like they did with the davinci releases of GPT3, GPT3.5, etc.
Seems like they're choosing Option 2. Long Microsoft runway
@damienriehl @ltmccarty how strong are the open source competitors? I'd rather not pay and use open source. But what I want to do seems at the outer edges of what even GPT4 can manage. Do the alternatives hold a candle?
Here is another article on those comments by Sam Altman:
https://www-wired-com.cdn.ampproject.org/c/s/www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/amp
@ltmccarty @damienriehl That's evidence of a plateau. Thanks for sharing.
@ltmccarty @lexpedite Yes, that's really helpful. Thanks, Thorne.
I wonder if this comes from the lack of high quality data sources. There are only so many human-created words. Reddit will only get you so far.
Last bastion of high quality legal data: Law? Judicial, statutory, and regulatory text seems like an evergreen source.
@damienriehl @lexpedite
My guess is: Both algorithms and data. They have probably been running tests beyond the GPT-4 horizon and seeing a sigmoid.
But it is infuriating that we have to guess like this, since they haven't disclosed any technical information about GPT-4.
My guess is: Both algorithms and data. They have probably been running tests beyond the GPT-4 horizon and seeing a sigmoid.
But it is infuriating that we have to guess like this, since they haven't disclosed any technical information about GPT-4.
On guessing and nondisclosure: Hard to be transparent when (1) for profit and (2) open source competitors are nipping at your heels.
Google can afford to be open. OpenAI (an oxymoron) apparently thinks that it can't.
Another consideration: Malfeasance and misuse of the model. Regulatory concerns abound.
@damienriehl @lexpedite
You are correct, of course. I am just unhappy about the current state of scientific research in this field.
https://arxiv.org/abs/2304.06035
You are correct, of course. I am just unhappy about the current state of scientific research in this field.
https://arxiv.org/abs/2304.06035
Choose Your Weapon: Survival Strategies for Depressed AI Academics
Are you an AI researcher at an academic institution? Are you anxious you are not coping with the current pace of AI advancements? Do you feel you have no (or very limited) access to the computational and human resources required for an AI research breakthrough? You are not alone; we feel the same way. A growing number of AI academics can no longer find the means and resources to compete at a global scale. This is a somewhat recent phenomenon, but an accelerating one, with private actors investing enormous compute resources into cutting edge AI research. Here, we discuss what you can do to stay competitive while remaining an academic. We also briefly discuss what universities and the private sector could do improve the situation, if they are so inclined. This is not an exhaustive list of strategies, and you may not agree with all of them, but it serves to start a discussion.
@damienriehl @ltmccarty I would be reluctant to categorize judicial writing as high quality for general purposes. It's not even high quality data for legal purposes in predicting the outcome of cases, because it is biased toward cases where one or more of the parties is rich and/or crazy.
This is a "compared to what" and "what goal" problem.
Compared to Reddit and Twitter? Judicial writing is pretty high quality.
Gauging the law's current state (e.g., Roe v. Wade as no longer current law)? Pretty high quality.
Prediction? For that, is there *any* high quality source?
@damienriehl @ltmccarty We are talking for general purpose language models, and compared to literally anything No one wants an AI that will take 40 pages to explain something in esoteric language, for the benefit of the losing party, while hedging their bets against appeal. There is no use case except drafting judgments for which judgments are "good" data, and there they are good only if you don't care whether the judgment is correct. Nevergreen.
@lexpedite @damienriehl
The purpose of pre-training is not to make predictions for specific tasks, but to build some kind of a model of the concepts underlying the lexical items in the texts. At least, that's the current understanding of how LLMs work.
So it does not matter what the outcome is in a particular case.
The purpose of pre-training is not to make predictions for specific tasks, but to build some kind of a model of the concepts underlying the lexical items in the texts. At least, that's the current understanding of how LLMs work.
So it does not matter what the outcome is in a particular case.
@damienriehl @lexpedite
Damien: Here's a question for Fastcase/vLex.
Suppose you trained a model the size of GPT-4, from scratch, on just the Fastcase/vLex database. No Wikipedia, no Reddit, just legal texts, but including Docket Alarm, etc. Everything you have.
How many tokens? How would this compare to the other datasets that have been used for other models? Note that I am talking about pre-training, not fine-tuning.
I am sure you have done the calculations, or will be soon.
Damien: Here's a question for Fastcase/vLex.
Suppose you trained a model the size of GPT-4, from scratch, on just the Fastcase/vLex database. No Wikipedia, no Reddit, just legal texts, but including Docket Alarm, etc. Everything you have.
How many tokens? How would this compare to the other datasets that have been used for other models? Note that I am talking about pre-training, not fine-tuning.
I am sure you have done the calculations, or will be soon.
You should talk to John Nay about what he's planning to build. 😏
LLM researcher affiliated with NYU and Stanford:
https://law.stanford.edu/directory/john-nay/
https://arxiv.org/a/nay_j_1.html
He's cooking something that should be good.
@lexpedite
@damienriehl
OK. What is the evidence to the contrary? Since OpenAI has told us zilch about GPT-4, any claims that we might want to make about future systems are pure speculation. I have been saying this for quite a while.
Here is a comment that captures the situation accurately, in my opinion:
Jeff Bigham (@[email protected])
a remarkable thing about LLMs is i can't find anyone, novice or expert, proponent or critic, who seems to have a particularly good intuition for how these things will develop going forward. i think part of the reason is that there are so many moving parts and none of those parts are either transparent or intuitive -- unintuitive statistics of large data, massive human annotation/tuning/iteration efforts, huge limitations in evaluation, etc. GPT-5? nothing burger or another leap, who knows
@ltmccarty @damienriehl They are getting better. That's the evidence. It's just a trend, not a guarantee, but so what?
The idea that they have plateaued verges on the delusional. Human beings have never gotten anything that right that quickly.