Powerful open-source K2-65B LLM costs 35% less to train
Powerful open-source K2-65B LLM costs 35% less to train
MBZUAI announces new LLM under LLM360 framework in collaboration with Petuum
#UAE #LLMs - Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) has announced the K2-65B large language model (LLM), developed in collaboration with Petuum, and released under the LLM360 framework for creating open-source large language models. The university hopes that K2 will set a new standard for sustainable performance in LLMs. The new model was trained using 35% fewer resources than Llama 2 70B, but with performance that compares to Llama 3. K2-65B is one of several cutting-edge code releases announced under LLM360, that support the mission to enable community-owned AGI. MBZUAI also announced the LLM360 Research Suite, the LLM360 Developer Suite, the LLM360 Pretraining Suite, and the LLM360 Model Performance and Evaluation Collection.
SO WHAT? - The LLM360 framework was launched last December to help developers create open-source large language models, easier, faster and cheaper. Training LLMs can be a costly and complex task, with irregular model behaviours and unexpected issues. The LLM360 initiative aims to bring a new level of transparency to open source code, opening up the whole LLM training process, code, data and best practices to help developers fast track development efforts. The new K2-65B LLM released, together with the research resources, guides and tutorials for developers, significantly enhance LLM360's reference library for open-source developers.
Here are the key details of the LLM360 announcement:
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) has announced the launch of its latest large language model K2-65B, in collaboration with Petuum, and a number of other open-source resources under the LLM360 framework.
According to MBZUAI, K2-65B is a cost-effective open-source LLM that presents a blueprint for transparent open-source Artificial General Intelligence (AGI). K2's release is expected to enhance knowledge sharing, fundamental research, and advanced technological transfer, through reproducibility and transparency.
K2-65B was trained on 1.4 trillion tokens using 480 NVIDIA A100 Tensor Core GPUs inside of NVIDIA’s DGX Cloud, an AI platform for enterprise developers, optimised for the demands of generative AI.
Training the new LLM used 35% fewer resources than Llama 2 70B, making it one of the world’s most sustainable models in its class. In certain functions, such as mathematical and logical reasoning, K2 is competitive against much larger models, such as GPT4.
K2-65B was trained in two stages and has demonstrated on par reasoning and text generation capabilities with strong domain knowledge in medicine, coding, and math. See LLM360’s technical report here.
The new model underwent rigorous evaluation through 22 multidisciplinary assessments, across various domains including math, coding, and medicine, among others, with the new model surpassing Llama 2 70B across each of these areas. According to MBZUAI, the chat model, K2-Chat, also outperforms Llama 2 70B Chat in every aspect of evaluation, underscores its capabilities in understanding and generating human-like responses across diverse scenarios.
The model is available globally under the Apache 2.0 licence.
The other code and information resources released are:
LLM360 Research Suite - This comprehensive set of LLM artifacts from Amber-7B, CrystalCoder-7B, and K2-65B models for academic and industry researchers to explore LLM training dynamics.
LLM360 Developer Suite - A series finetuning and inference tutorials for Amber-7B, CrystalCoder-7B, and K2-65B models for tech enthusiasts, AI practitioners and academic or industry who are interested in general model usage or downstream task evaluation and research.
LLM360 Pretraining Suite - a series of step-by-step guides to reproduce Amber-7B, CrystalCoder-7B, and K2-65B models for tech enthusiasts, AI practitioners and academic or industry researchers to transfer knowledge on LLM pretraining techniques.
LLM360 Model Performance and Evaluation Collection - robust large language model evaluation suite consisting of general and domain specific evaluations to assess model knowledge and function.
Read more about LLM360
New framework for open-source LLMs (Middle East AI News)