NanoGPT Slowrun - Q

Python Trending 🇺🇦 (@pythontrending) on X

dllm - dLLM: Simple Diffusion Language Modeling https://t.co/C1OuWPVlg2

Tokenization for language modeling: Byte Pair Encoding vs Unigram Language Modeling

Tokenizers used by the best-performing language models (Bert, GPT-2, etc.) poorly reflect the morphology of English text. I had hoped to use some quarantine time to design one that more closely aligns to relationships between wordforms. But Kaj Bostrom and Greg Durrett beat me to it and so this blog post materialized instead. I add some additional motivation, evaluate both methods against ‘gold standard’ tokenizations, and speculate about what might come next.

Andrey Markov & Claude Shannon Counted Letters to Build the First Language-Generation Models

Shannon’s said: “OCRO HLI RGWR NMIELWIS”

GitHub - edwko/OuteTTS: Interface for OuteTTS models.

Interface for OuteTTS models. Contribute to edwko/OuteTTS development by creating an account on GitHub.

Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs

Most interpretability research in NLP focuses on understanding the behavior and features of a fully trained model. However, certain insights into model behavior may only be accessible by observing the trajectory of the training process. We present a case study of syntax acquisition in masked language models (MLMs) that demonstrates how analyzing the evolution of interpretable artifacts throughout training deepens our understanding of emergent behavior. In particular, we study Syntactic Attention Structure (SAS), a naturally emerging property of MLMs wherein specific Transformer heads tend to focus on specific syntactic relations. We identify a brief window in pretraining when models abruptly acquire SAS, concurrent with a steep drop in loss. This breakthrough precipitates the subsequent acquisition of linguistic capabilities. We then examine the causal role of SAS by manipulating SAS during training, and demonstrate that SAS is necessary for the development of grammatical capabilities. We further find that SAS competes with other beneficial traits during training, and that briefly suppressing SAS improves model quality. These findings offer an interpretation of a real-world example of both simplicity bias and breakthrough training dynamics.

Bevor Sie zu Google Maps weitergehen

GPT-4 API by OpenAI Now Available – Analytics India Magazine #GPT4API

OpenAI announces general availability for GPT-4 API and plans to remove older models in Completion API by next yearOpenAI announces general availability for GPT-4 API and plans to remove older models in Completion API