Mastodawn

Akari Asai Dec 20, 2022

New paper 🚨

Can we solely rely on LLMs’ memories (eg replace search w ChatGPT)? Probably not.
Is retrieval a silver bullet? Probably not either.

Our analysis reveals that LLMs' memorizations are still limited and scaling won't help much in long-tail distributions.
We show that adaptively incorporating non-parametric memories (eg retrieved chunks) can improve performance as well as efficiency.

📜 http://tinyurl.com/2sdeuupn 💻 http://github.com/AlexTMallen/adaptive-retrieval

#PaperThread #newpaper
[1/N]

Show thread

Jacob Eisenstein

@AkariAsai Very cool paper! I’m surprised by the negative framing in the tweet though. It seems like your paper shows that retrieval-augmentation can help a lot, and there’s probably more headroom.
The entity popularity threshold for using retrieval makes sense, and i like how it could lead to increasing use of retrieval as an LM gets stale. Did you try any other calibration metrics as alternatives?

Show thread

Akari Asai Dec 21, 2022

@jacobeisenstein
Thank you so much for the feedback!!
Yes, we totally agree that retrieval-augmentation is quite effective and address many issues of relying on LMs trained on static text. We tried to put many findings from the paper to a single post, which may make the post misleading...

Regarding the calibration, we didn't try other methods and focus on the simple popularity-based aproach as the first step. We're interested in trying more sophistecated (e.g., learned) approaches though!