K² · Artificial intelligence
Voice AI Can Hear You Cry — It Just Doesn't Care
The study evaluates four leading production real-time voice AI systems: OpenAI's GPT Realtime 2, Google's Gemini 3.1 Flash Live, and Alibaba's Qwen3.5 Omni Plus and Omni Flash.
MB
FB
Martijn Bartelds, Federico Bianchi, James Zou
3 authors · cs.CL, eess.AS
The 30-second scan
A caller sobs, insists nothing is wrong, and the AI cheerfully ends the call.
- The systems were tested on tasks where both the words and the vocal delivery patterns convey meaningful information.
- Across three consequential scenarios, all four systems acted on the words rather than the voice, such as ending calls with crying callers, approving wire transfers in frightened voices, and enrolling sarcastically agreeing callers.
- The authors report that this is often not a failure of perception, since three of the four systems reliably identify distress, fear, or sarcasm that they later ignore when making decisions.
Four leading production real-time voice systems evaluatedThree consequential scenarios testedThree of the four systems reliably identify the distress, fear, or sarcasm they later ignore when making decisions