- Tests 24 Malaysian consumer chatbots across four sectors
- Two-thirds don’t understand Bahasa Melayu, Manglish, or both, a glaring shortcoming
Entermind, a data and AI consultancy, today announced the launch of Confessions of an AI Chatbot, a whitepaper revealing the findings of Malaysia’s first systematic study of customer service chatbot quality across travel,telecommunications, eCommerce & Wallet, and financial services.
The findings are surely to cause some emergency meetings among the companies who fared poorly. “I think this is an accurate assesment!” exclaimed the CEO of one company whose bot was poorly rated.
Entermind, which is headquartered in Kuala Lumpur with offices in Singapore, Bangalore, and San Francisco, noted that Malaysia is investing billions in AI infrastructure. Yet, when a real customer (as opposed to an AI bot) opens a chatbot to cancel a flight, dispute a bill, or check their balance, the experience tells a very different story.
Drawing on the Entermind Chatbot Quality Index (CQI), a proprietary framework for the systematic testing and quality assessment of customer service chatbots, the study evaluated 24 of the country’s top customer service chatbots against 26 standardised binary tests grouped across five weighted categories: Comprehension, Access, Experience, Functional Capability, and Safeguards.
The findings reveal a market in which a small group of chatbots are genuinely delivering, and the majority are quietly failing the people they were built to serve.
What the data found
The average CQI score across all 24 chatbots is 49.5%. Eleven of the 24 score below 40%. Six do nothing more than greet users and route them to a human agent. Only three can complete a transaction within the chat itself. The lowest score in the entire index? 4.0%.
The comprehension gap is particularly stark. Only 23% of chatbots correctly understand negation. Only 30% can handle a change of topic mid-conversation. Two-thirds cannot understand Bahasa Melayu, Manglish, or both. In a market of 34 million people where multilingual communication is the norm, this is a glaring shortcoming.
“Most chatbots in Malaysia are a glorified FAQ search bar,” said one telco evaluator whose feedback is featured in the whitepaper.
.jpg)
The investment gap
The whitepaper identifies capability investment and design as the two factors that separate the top performers from the rest. Chatbots scoring above 70%, seven of the 24 tested, share these common traits: genuine language-understanding, end-to-end
transaction architecture, and robust safety infrastructure. Those scoring below 40% typically fail across all these dimensions simultaneously.
Commenting on the whitepaper, Prashant Kumar (pic), founder and CEO of Entermind, said, “GPTs (Generative Pre-trained Transformers) are rapidly resetting expectations that customers have from enterprise AI bots. It’s clear from the audit that the gap is yawning. As Generation AI shapes up, the upsides are highly rewarding.”
Related Articles
Keyword(s) :