AI Conference Deadlines

Countdowns to top AI/ML/CV/NLP/robotics conference deadlines

I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Token

Large Language Models are known to capture real-world knowledge, allowing them to excel in many downstream tasks. Despite recent advances, these models are still prone to what are commonly known as hallucinations, causing them to emit unwanted and factually incorrect text. In this work, we propose a novel calibration method that can be used to combat hallucinations. We add a special [IDK] ("I don't know") token to the model's vocabulary and introduce an objective function that shifts probability mass to the [IDK] token for incorrect predictions. This approach allows the model to express uncertainty in its output explicitly. We evaluate our proposed method across multiple model architectures and factual downstream tasks. We find that models trained with our method are able to express uncertainty in places where they would previously make mistakes while suffering only a small loss of encoded knowledge. We further perform extensive ablation studies of multiple variations of our approach and provide a detailed analysis of the precision-recall tradeoff of our method.

AI@HPI Conference | Hasso-Plattner-Institut

GitHub - BleTib/artquest

Contribute to BleTib/artquest development by creating an account on GitHub.

Efficient Parallelization Layouts for Large-Scale Distributed Model Training

Efficiently training large language models requires parallelizing across hundreds of hardware accelerators and invoking various compute and memory optimizations. When combined, many of these strategies have complex interactions regarding the final training efficiency. Prior work tackling this problem did not have access to the latest set of optimizations, such as FlashAttention or sequence parallelism. In this work, we conduct a comprehensive ablation study of possible training configurations for large language models. We distill this large study into several key recommendations for the most efficient training. For instance, we find that using a micro-batch size of 1 usually enables the most efficient training layouts. Larger micro-batch sizes necessitate activation checkpointing or higher degrees of model parallelism and also lead to larger pipeline bubbles. Our most efficient configurations enable us to achieve state-of-the-art training efficiency results over a range of model sizes, most notably a Model FLOPs utilization of 70.5% when training a Llama 13B model.

near on X

fine-tuning cautionary tale

AI Essentials for Decision Makers - Navigating the future of Business

Jährlich besuchen zahlreiche Fach- und Führungskräfte die Workshops und Programme der HPI Academy, um sich weiterzubilden oder um ihre erworbenen Kompetenzen in den Bereichen Design Thinking, Leadership, Innovation & Transformation sowie IT & Digitalisierung zertifizieren zu lassen. Wir garantieren Ihnen höchste Qualitätsstandards und beste Lernerlebnisse sowohl für unsere Präsenz- als auch unsere Online-Angebote. Lassen Sie sich also inspirieren, bei uns vor Ort oder von überall auf der Welt. Sehr gerne unterstützen wir Sie auf Ihrer persönlichen Lernreise.

Urania Verein - Künstliche Intelligenz - Chance oder Bedrohung?

Managing extreme AI risks amid rapid progress