New framework for open-source LLMs

LLM360 helps reduce complexity and size to enable new open-source LLMs

#UAE #LLLMs - Californian AI supercomputer developer Cerebras Systems, generative AI company Petuum, and Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) have launched a new framework for creating open-source large language models. Called LLM360, the framework allows developers to create LLMs easier, faster and cheaper by empowering them with detailed insights and methodologies. The LLM360 framework comes with a commitment to open up the whole LLM training process, code, data and best practices to empower developers to fast track their own development efforts.

SO WHAT? - Open source code is available for a growing number of large language models, giving developers free access to see the models' code and architecture, allowing them to modify models and to fine-tune pre-trained models using their own different domain-specific knowledge and training data. However, training LLMs can be a highly complex task, with irregular model behaviours and unexpected issues that require expert knowledge to fix. The LLM360 initiative promises to share models, training process, code, data and insights for each LLM, allowing developers to understand all the nuances of model development.

• The LLM360 announcement follows the announcement of Mohamed bin Zayed University of Artificial Intelligence's Institute of Foundation Models, and its goal of helping to democratise AI research.

• A collaboration between Petuum, MBZUAI and Cerebras, LLM360 aims to empower the open-source large language model community with more transparency, more knowledge and more insight into AI models, to help enable trustworthy and innovative research.

• The LLM360 initiative launches with two open source large language models both using Meta’s LLaMA architecture: Amber, a 7 billion parameter English-language model trained on 1.2 trillion tokens; and CrystalCoder, a new 7 billion parameter model designed for English language and coding tasks. Diamond, a third 65 billion parameter model will be released soon.

• Developers and researchers can access detailed insights into the Amber and CrystalCoder LLMs, including the code, the pretraining, full data sequences, source code, and detailed logs.

• Both of the new LLM360 models were trained on Condor Galaxy 1, the AI supercomputer built by G42 and Cerebras Systems.

ZOOM OUT - There is growing demand for open source AI models, as more and more developers and end-user organisations seek alternatives to proprietary 'black-box' models that may carry hidden risks. The hidden characteristics of proprietary AI models make it difficult for large organisations to assess business risks associated with algorithm logic, safety, plus ethical and legal concerns. Meanwhile, it appears that the European Union's new law governing the use of artificial intelligence focuses on proprietary models and may have fewer restrictions on open source models.

However, many of the highest quality and best performing LLMs are, in fact, proprietary and the fact that a big brand or research lab puts its name to an AI model is seen as a reference for quality. Therefore, improving the overall quality of open source LLMs and empowering more developers to adopt the latest techniques and best practices is definitely in the interests of both the open source community and customers that are leaning towards open-source AI models.

IMO - The launch of the LLM360 framework brings even more focus to the role of Abu Dhabi's AI ecosystem, which has garnered global attention with the release of the Falcon LLM by Technology Innovation Institute (TII) and the Arabic-language model Jais from Core42, both of which are available as open source.

There are relatively few serious contenders at present to provide enterprise-ready open source AI models, which represents a significant regional and global opportunity for Abu Dhabi's developers. The LLM360 initiative could bring more global developers closer to Abu Dhabi's and help further establish the ecosystem as a global hub for open source AI model development.

LINKS

• See the LLM360 website (includes Hugging Face & Github links to models)

• LLM360 on Github

• Read the research paper on LLM360 (PDF download)

Also read:

• Abu Dhabi launches new global AI company (Middle East AI News)

• TII announces Falcon 180B LLM (Middle East AI News)

• Cerebras & G42 build massive cloud supercomputer network (Middle East AI News)

• Will GenAI champion the Arabic language? (Middle East AI News)