Mury Fajar Dewantoro, Febri Abdullah, Yi Xia, Ibrahim Khan, Ruck Thawonmas, Wenwen Ouyang, Fitra Abdurrachman Bachtiar
This study extends our previous investigation into whether multimodal large language models (MLLMs) can reason about physical reasoning, using a game environment as the testbed. Stability served as a foundational scenario to probe model understanding of physical reasoning. We evaluated twelve models, combining those from the earlier study with six additional open-weight models, across three tasks designed to capture different aspects of reasoning. Human participants were included as a reference point, consistently achieving the highest accuracy, underscoring the gap between model and human performance. Among MLLMs, the GPT series continued to perform strongly, with GPT-4o showing reliable results in image-based tasks, while the Qwen2.5-VL series reached the highest overall scores in this extended study and in some cases surpassed commercial counterparts. Simpler binary tasks yielded balanced performance across modalities, suggesting that models can capture certain basic aspects of reasoning, whereas more complex multiple-choice tasks led to sharp declines in accuracy. Structured inputs such as XML improved results in the prediction task, where Qwen2.5-VL outperformed GPT variants in our earlier work. These findings demonstrate progress in scaling and modality design for physical reasoning, while reaffirming that human participants remain superior across all tasks. © 2025 by the authors.
Graduate School of Information Science and Engineering, Ritsumeikan University, 2-150 Iwakura-cho, Osaka, Ibaraki, 567-8570, Japan; Center for Information and Neural Networks, National Institute of Information and Communications Technology, Osaka, Suita, 565-0871, Japan; College of Information Science and Engineering, Ritsumeikan University, 2-150 Iwakura-cho, Osaka, Ibaraki, 567-8570, Japan; Alumni of College of Engineering, Electrical and Computer Engineering Department, Carnegie Mellon University, Pittsburgh, 15213, PA, United States; Faculty of Computer Science, Universitas Brawijaya, Malang, 65145, Indonesia
