My life goal is to make a comic book for the comedy gold my mom spins into my life.
And a graphic novel for all the ingots of tragedy. 🤣
My mom is in parent time-out for the following dialogue during work this week:
Mom: Jonathan, we need to talk
Me: what’s wrong are you okay?
Mom: do you have sea salt?
Me: I, what?
Mom: SEA SALT, LIKE SALT BUT FROM THE SEA
Me: …uh yeah? But I’m at work, can we—
Mom: Get KOREAN sea salt.
Me: okay, sure!
Mom: Did you just say *sure* to your *mother*?!
Me: -internally- oh shit
Mom: Well, that’s fine, WHEN IM DEAD no one will bother you about sea salt! *hangs up*
A recent trend is to fine-tune open-source LMs on ChatGPT outputs (e.g., Alpaca, Self-Instruct, Vicuna), with the aim of broadly imitating the model. In our new paper, we critically analyze this approach.
An emerging method to cheaply improve a weaker language model is to finetune it on outputs from a stronger model, such as a proprietary system like ChatGPT (e.g., Alpaca, Self-Instruct, and others). This approach looks to cheaply imitate the proprietary model's capabilities using a weaker open-source model. In this work, we critically analyze this approach. We first finetune a series of LMs that imitate ChatGPT using varying base model sizes (1.5B--13B), data sources, and imitation data amounts (0.3M--150M tokens). We then evaluate the models using crowd raters and canonical NLP benchmarks. Initially, we were surprised by the output quality of our imitation models -- they appear far better at following instructions, and crowd workers rate their outputs as competitive with ChatGPT. However, when conducting more targeted automatic evaluations, we find that imitation models close little to none of the gap from the base LM to ChatGPT on tasks that are not heavily supported in the imitation data. We show that these performance discrepancies may slip past human raters because imitation models are adept at mimicking ChatGPT's style but not its factuality. Overall, we conclude that model imitation is a false promise: there exists a substantial capabilities gap between open and closed LMs that, with current methods, can only be bridged using an unwieldy amount of imitation data or by using more capable base LMs. In turn, we argue that the highest leverage action for improving open-source models is to tackle the difficult challenge of developing better base LMs, rather than taking the shortcut of imitating proprietary systems.
Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved With Text
103M documents containing 585M images interleaved with 43B English tokens
MultimodalC4 is a multimodal extension of c4 that interleaves millions of images with text. - GitHub - allenai/mmc4: MultimodalC4 is a multimodal extension of c4 that interleaves millions of images...
Learning to Tokenize for Generative Retrieval
Establishes the new state-of-the-art on the NQ320K dataset via better generative retrieval.
Conventional document retrieval techniques are mainly based on the index-retrieve paradigm. It is challenging to optimize pipelines based on this paradigm in an end-to-end manner. As an alternative, generative retrieval represents documents as identifiers (docid) and retrieves documents by generating docids, enabling end-to-end modeling of document retrieval tasks. However, it is an open question how one should define the document identifiers. Current approaches to the task of defining document identifiers rely on fixed rule-based docids, such as the title of a document or the result of clustering BERT embeddings, which often fail to capture the complete semantic information of a document. We propose GenRet, a document tokenization learning method to address the challenge of defining document identifiers for generative retrieval. GenRet learns to tokenize documents into short discrete representations (i.e., docids) via a discrete auto-encoding approach. Three components are included in GenRet: (i) a tokenization model that produces docids for documents; (ii) a reconstruction model that learns to reconstruct a document based on a docid; and (iii) a sequence-to-sequence retrieval model that generates relevant document identifiers directly for a designated query. By using an auto-encoding framework, GenRet learns semantic docids in a fully end-to-end manner. We also develop a progressive training scheme to capture the autoregressive nature of docids and to stabilize training. We conduct experiments on the NQ320K, MS MARCO, and BEIR datasets to assess the effectiveness of GenRet. GenRet establishes the new state-of-the-art on the NQ320K dataset. Especially, compared to generative retrieval baselines, GenRet can achieve significant improvements on the unseen documents. GenRet also outperforms comparable baselines on MS MARCO and BEIR, demonstrating the method's generalizability.
unarXive 2022: All arXiv Publications Pre-Processed for NLP, Including Structured Full-Text and Citation Network
abs: https://arxiv.org/abs/2303.14957
repo: https://github.com/IllDepence/unarXive
Large-scale data sets on scholarly publications are the basis for a variety of bibliometric analyses and natural language processing (NLP) applications. Especially data sets derived from publication's full-text have recently gained attention. While several such data sets already exist, we see key shortcomings in terms of their domain and time coverage, citation network completeness, and representation of full-text content. To address these points, we propose a new version of the data set unarXive. We base our data processing pipeline and output format on two existing data sets, and improve on each of them. Our resulting data set comprises 1.9 M publications spanning multiple disciplines and 32 years. It furthermore has a more complete citation network than its predecessors and retains a richer representation of document structure as well as non-textual publication content such as mathematical notation. In addition to the data set, we provide ready-to-use training/test data for citation recommendation and IMRaD classification. All data and source code is publicly available at https://github.com/IllDepence/unarXive.
Walking through ohare and giggling about this.
Several stupid bits:
1) raccoon dives in, has best garbage experience of its life
2) I dive in, have best garbage experience of my life
3) a cannabis confessional is set up next to it
The list goes on
kasdeya
Aran Komatsuzaki
Ted Underwood