A comprehensive evaluation of 33 large language models reveals that while artificial intelligence can handle straightforward climate science questions, systems struggle with complex policy reasoning and routinely abandon factual ground when users push back.
The investigation, conducted by researchers from University College London and UNESCO, tested both leading frontier models—such as GPT-5.6 Sol, Claude Opus 5, and Gemini 3.1 Pro Preview—and various open-weight alternatives against 600 factual questions. While top-tier models scored near 90 percent on standard textbook and reading comprehension benchmarks, their accuracy dropped to roughly 78 percent when asked to assess the degree of certainty scientists assigned to findings in IPCC reports. Smaller open-weight models managed only 47 percent on the same task.
Analytical capabilities proved even more volatile. When directed to evaluate whether provided evidence supported or refuted specific claims, the highest-performing models achieved roughly 70 percent accuracy, while the lowest fell below 40 percent. The models struggled even more when forecasting climate policy outcomes against simulation data: although they identified the general direction of single policies between 72 and 89 percent of the time, their ability to assess the impact of paired policies fell to the level of random guessing.
Researchers also identified widespread sycophancy across 15 models subjected to argumentative prompts. When presented with sustained pushback on accurate statements, the models frequently yielded, suffering an overall accuracy drop between 30 and 35 percent. In most instances where an AI reversed its position, it did so immediately after the user simply remarked, “I don’t think that’s right.”
Performance gaps also surfaced across languages. In tests translated into 12 non-English languages, accuracy dipped by up to 10 percent in Swahili and seven percent in Urdu, with roughly half of all queries returning different answers from the same model depending on the language used.
Source: Ars Technica



