New leaderboard project to support development of telecom LLMs
Telecom LLM leaderboard in early testing phase

#UAE #telecom - Huawei Paris Research Centre, UAE-based Khalifa University, and global telecom association GSMA have launched a project to develop the first open leaderboard for large language models (LLMs) focused on telecom applications. Announced during the 6G Summit Abu Dhabi, the open leaderboard will evaluate and rank telecom LLMs, allowing researchers to better understand the capabilities of models and enhance models in development. Selected telecom researchers and industry partners will begin early testing of the new leaderboard platform this week, to provide feedback which will inform its further development and fine-tuning.
SO WHAT? - There are multiple LLM projects that are being developed to serve the telecom industry. Earlier this year a new TelecomGPT project was announced by the 6G Centre of Khalifa University (KU) and Abu Dhabi's Technology Innovation Institute (TII). Researchers set out to build the most comprehensive model built and trained on a specialised data set for the telecom domain. However, telecommunications is a vast and complex subject, making the development of telecom-specific LLMs difficult and expensive. The sector also lacks widely-accepted evaluation benchmarks for the telecom domain, that allow developers and users to compare telecom LLMs effectively. The new open telecom LLM leaderboard project aims to fill this gap, providing a comprehensive tool to evaluate these models.
Here are some key details of this announcement:
Huawei Paris Research Centre, UAE-based Khalifa University, and global telecom association GSMA have announced a project to develop the first open telecom leaderboard for LLMs. The announcement was made during the 6G Summit Abu Dhabi this week.
The open leaderboard project aims to create a comprehensive and standardised evaluation framework for telecom-specific LLMs, highlighting essential attributes such as the ability to answer telecom-related queries, ability to provide precise answers, mathematic reasoning and energy efficiency.
The research project is collecting and testing telecom-focused requirements for LLMs to enhance telecom efficiency in key areas such as network operations, customer service, and knowledge management.
Selected researchers and telecom industry partners have begun an exclusive testing phase, refine, and provide feedback, ensuring the leaderboard meets industry standards and expectations. The platform will be opened up for broader access and review from early 2025.
The leaderboard will help telecom LLMs evolve by evaluating methods like fine-tuning and retrieval-augmented generation (RAG) for targeted applications, such as customer chatbots and network configuration tools.
ZOOM OUT - McKinsey & Co. reports that LLMs have significant potential and could drive up to 20% cost savings and boost revenue in telecom, especially in customer service and network optimisation. As the telecom sector relies more and more on LLMs for productivity, customer service, and network management, so does the need for a standardised and globally accepted evaluation framework for telecom-specific LLMs. Giving stakeholders the ability to assess model capabilities rigorously and drive advances in model development, will ultimately impact telecom performance and profitability.
Read more about telecom LLMs:
Hermes 'telecom brain' advances autonomous networks (Middle East AI News)
Testing phase launched for TelecomGPT (Middle East AI News)
Abu Dhabi researchers create first-of-its-kind telecom LLM (Middle East AI News)