K² · Artificial intelligence
Teaching image-reading AI to stop and run code when the math gets hard
The paper introduces AIR, a method that empowers multimodal large language models (MLLMs) with adaptive interleaved reasoning capabilities using code.
CH
XL
Cong Han, Xiaohan Lan, Haibo Qiu et al.
4 authors · cs.CV, cs.AI
The 30-second scan
Ask an AI that reads pictures to do real arithmetic, and it often fails.
- The approach uses extended reinforcement learning training on code-augmented complex numerical computation tasks.
- The authors argue existing tool-use approaches rely on predefined heuristics for visual manipulation and cannot address numerical computation problems due to their exclusive focus on visual operations.
- The proposed solution has three components: a two-stage cold-start data construction pipeline, data filtering strategies for RL dataset curation, and an adaptive tool-invocation strategy using a group-constrained reward function.
Performance improves by an average of 6.1 percentage points on evaluation benchmarks after RL training with the group-constrained reward functionAccuracy for interleaved reasoning samples increases by 9.9 percentage pointsOverall success rate of tool-use exceeds 95%