Mastodawn

Jason Morris Apr 13, 2023

I think this #FutureLaw 2023 panel on GPT4 is good in terms of a balanced view of the risks and opportunities of using tools like GPT4. What surprises me is that no one is talking about how soon GPT4 will be obsolete, and replaced by something that improves over the previous iteration just as significantly. #LegalTech #LawFedi

L Thorne McCarty Apr 14, 2023

@lexpedite

Some evidence that GPTs may have reached a plateau?

https://www.theverge.com/2023/4/14/23683084/openai-gpt-5-rumors-training-sam-altman

OpenAI’s CEO confirms the company isn’t training GPT-5 and ‘won’t for some time’

OpenAI’s CEO Sam Altman has confirmed that the company is not currently training GPT-5 — the successor to its language model GPT-4, released this March. Altman was discussing fears about AI safety.

The Verge

Jason Morris Apr 14, 2023

@ltmccarty that article just says you can't assume that new versions are better than earlier versions in any stable degree. Granted. But no one is assuming. It is objectively getting significantly better. That they haven't started "training" GPT5 yet doesn't mean anything. I don't see any evidence of a plateau, and I see lots of evidence to the contrary.

@lexpedite @ltmccarty — Two ways this could play out:

1. OpenAI feels threatened by the many free + open-source competitors (e.g., Eleuther, Dolly-2), and wonders whether the time/expense of generating a Foundational Model is worth it — when they're competing with "free."

2. OpenAI — with Microsoft money — takes a run at improving the existing GPT-4 model incrementally. Like they did with the davinci releases of GPT3, GPT3.5, etc.

Seems like they're choosing Option 2. Long Microsoft runway

Jason Morris Apr 14, 2023

@damienriehl @ltmccarty how strong are the open source competitors? I'd rather not pay and use open source. But what I want to do seems at the outer edges of what even GPT4 can manage. Do the alternatives hold a candle?

L Thorne McCarty Apr 17, 2023

@damienriehl @lexpedite

Here is another article on those comments by Sam Altman:

https://www-wired-com.cdn.ampproject.org/c/s/www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/amp

OpenAI’s CEO Says the Age of Giant AI Models Is Already Over

Sam Altman says the research strategy that birthed ChatGPT is played out and future strides in artificial intelligence will require new ideas.

WIRED

Jason Morris Apr 18, 2023

@ltmccarty @damienriehl That's evidence of a plateau. Thanks for sharing.

Damien Riehl Apr 18, 2023

@ltmccarty @lexpedite Yes, that's really helpful. Thanks, Thorne.

I wonder if this comes from the lack of high quality data sources. There are only so many human-created words. Reddit will only get you so far.

Last bastion of high quality legal data: Law? Judicial, statutory, and regulatory text seems like an evergreen source.

L Thorne McCarty Apr 18, 2023

@damienriehl @lexpedite

My guess is: Both algorithms and data. They have probably been running tests beyond the GPT-4 horizon and seeing a sigmoid.

But it is infuriating that we have to guess like this, since they haven't disclosed any technical information about GPT-4.

Damien Riehl Apr 18, 2023

@ltmccarty @lexpedite

On guessing and nondisclosure: Hard to be transparent when (1) for profit and (2) open source competitors are nipping at your heels.

Google can afford to be open. OpenAI (an oxymoron) apparently thinks that it can't.

Damien Riehl Apr 18, 2023

@ltmccarty @lexpedite

Another consideration: Malfeasance and misuse of the model. Regulatory concerns abound.

L Thorne McCarty Apr 18, 2023

@damienriehl @lexpedite

You are correct, of course. I am just unhappy about the current state of scientific research in this field.

https://arxiv.org/abs/2304.06035

Choose Your Weapon: Survival Strategies for Depressed AI Academics

Are you an AI researcher at an academic institution? Are you anxious you are not coping with the current pace of AI advancements? Do you feel you have no (or very limited) access to the computational and human resources required for an AI research breakthrough? You are not alone; we feel the same way. A growing number of AI academics can no longer find the means and resources to compete at a global scale. This is a somewhat recent phenomenon, but an accelerating one, with private actors investing enormous compute resources into cutting edge AI research. Here, we discuss what you can do to stay competitive while remaining an academic. We also briefly discuss what universities and the private sector could do improve the situation, if they are so inclined. This is not an exhaustive list of strategies, and you may not agree with all of them, but it serves to start a discussion.

arXiv.org

Jason Morris Apr 18, 2023

@damienriehl @ltmccarty I would be reluctant to categorize judicial writing as high quality for general purposes. It's not even high quality data for legal purposes in predicting the outcome of cases, because it is biased toward cases where one or more of the parties is rich and/or crazy.

Damien Riehl Apr 18, 2023

@lexpedite @ltmccarty

This is a "compared to what" and "what goal" problem.

Compared to Reddit and Twitter? Judicial writing is pretty high quality.

Gauging the law's current state (e.g., Roe v. Wade as no longer current law)? Pretty high quality.

Prediction? For that, is there *any* high quality source?

Jason Morris Apr 18, 2023

@damienriehl @ltmccarty We are talking for general purpose language models, and compared to literally anything No one wants an AI that will take 40 pages to explain something in esoteric language, for the benefit of the losing party, while hedging their bets against appeal. There is no use case except drafting judgments for which judgments are "good" data, and there they are good only if you don't care whether the judgment is correct. Nevergreen.

L Thorne McCarty Apr 18, 2023

@lexpedite @damienriehl

The purpose of pre-training is not to make predictions for specific tasks, but to build some kind of a model of the concepts underlying the lexical items in the texts. At least, that's the current understanding of how LLMs work.

So it does not matter what the outcome is in a particular case.

L Thorne McCarty Apr 18, 2023

@damienriehl @lexpedite

Damien: Here's a question for Fastcase/vLex.

Suppose you trained a model the size of GPT-4, from scratch, on just the Fastcase/vLex database. No Wikipedia, no Reddit, just legal texts, but including Docket Alarm, etc. Everything you have.

How many tokens? How would this compare to the other datasets that have been used for other models? Note that I am talking about pre-training, not fine-tuning.

I am sure you have done the calculations, or will be soon.

Damien Riehl Apr 18, 2023

@ltmccarty @lexpedite

You should talk to John Nay about what he's planning to build. 😏

L Thorne McCarty Apr 18, 2023

@damienriehl @lexpedite

Sorry, I don't know him. Where is he?

Damien Riehl Apr 18, 2023

@ltmccarty @lexpedite

LLM researcher affiliated with NYU and Stanford:
https://law.stanford.edu/directory/john-nay/

https://arxiv.org/a/nay_j_1.html

He's cooking something that should be good.

John Nay | Stanford Law School

John Nay is an A.I. researcher and the co-founder and CEO of an A.I. technology company, Brooklyn Artificial Intelligence Research (Skopos Labs, Inc.)

Stanford Law School