"In LRMs, the term “reasoning” seems to be equated with generating plausible-sounding natural-language steps to solving a problem, and the extent to which this provides general and interpretable problem-solving abilities is still an open question. The performance of these models on math, science, and coding benchmarks is undeniably impressive. However, the overall robustness of their performance remains largely untested, especially for reasoning tasks that, unlike those the models were tested on, don’t have clear answers or cleanly defined solution steps, which is the case for many, if not most, real-world problems, not to mention “ fixing the climate, establishing a space colony, and the discovery of all of physics,” which are achievements OpenAI’s Sam Altman expects from AI in the future. And although LRMs’ chains of thought are touted for their “human interpretability,” it remains to be determined how faithfully these generated natural-language “thoughts” represent what is actually going on inside the neural network in the process of solving a problem. Multiple studies (carried out before the advent of LRMs) have shown that when LLMs generate explanations for their reasoning, the explanations are not always faithful to what the model is actually doing.
Moreover, the anthropomorphic language used in these models may mislead users into trusting them too much. The problem-solving steps that LRMs generate are often referred to as “thoughts”; the models themselves tell us that they are “thinking” (...) According to an OpenAI spokesperson, “Users have told us that understanding how the model reasons through a response not only supports more informed decision-making but also helps build trust in its answers.” But the question is, are users building trust based mainly on these humanlike touches, when the underlying model is less than trustworthy?"
https://www.science.org/doi/10.1126/science.adw5211
#AI #GenerativeAI #LLMs #ChatBots #LRMs #Reasoning