Samsung launches its own AI performance benchmark

Samsung was the first brand to introduce a comprehensive suite of AI features on smartphones. Since then, the company has been consistently launching new and improved AI features every six months. In an interesting move, Samsung has now launched its own AI performance benchmarking tool called TRUEBench.

Earlier today, Samsung announced that it has developed its own AI benchmarking tool called Trustworthy Real-world Usage Evaluation Benchmark (TRUEBench). It is a proprietary benchmark and was developed by Samsung Research after the company found gaps in existing AI benchmarking tools. Samsung says most existing tools only focus on English and are limited to single‑turn question‑answer structures.

Samsung says that TRUEBench has tests that incorporate diverse dialog scenarios and multilingual conditions. Based on its in-house use of AI for productivity, the benchmark evaluates AI performance based on ten of the most commonly used enterprise tasks, like content generation, data analysis, text summarization, and translation.

It has 2,485 test sets spread across 10 categories, 46 sub-categories, and 12 languages. It tests what AI models can actually solve and has test sets ranging from 8 characters to over 20,000 characters, reflecting tasks ranging from simple ones to long document summarization.

TRUEBench is claimed to have a reliable scoring system, thanks to an AI-powered automatic evaluation system that was collaboratively designed and refined by AI and humans. Its data samples and leadership boards are available on Hugging Face, which is an open-source platform. It allows people to test up to five AI models and compare them for performance and efficiency.

Paul (Kyungwhoon) Cheun, CTO of the DX Division at Samsung Electronics and Head of Samsung Research, said, “Samsung Research brings deep expertise and a competitive edge through its real-world AI experience. We expect TRUEBench to establish evaluation standards for productivity and solidify Samsung’s technological leadership.“