TII announces Falcon 180B LLM
TII releases Falcon 180B: an LLM 2.5 times bigger than Meta AI's Llama 2
#UAE #LLMs - Technology Innovation Institute (TII), the R&D lab of Abu Dhabi's Advanced Technology Research Council (ATRC), has released the world's most powerful open access large language model, Falcon 180B.
SO WHAT? - The announcement of Falcon 180B is a significant milestone in Abu Dhabi's AI research and development journey and proves that the fast-growing R&D hub can produce world leading computational research results.
Due to it being shared via Hugging Face, Falcon 180B is today the world's largest openly available LLM. The LLM is 2.5 times larger than Meta AI's Llama 2 and was trained with 4x more compute.
With 180 billion parameters the model was trained on 3.5 trillion tokens in a single epoch (i.e the algorithm worked through the entire training data set in one training cycle).
The production training for Falcon 180B was run on 4096 GPUs using the Amazon SageMaker cloud machine learning platform for a total of about 7,000,000 GPU hours.
ZOOM OUT - Abu Dhabi research labs have been developing LLM project for the past 2-3 years, with TII and G42's Inception and Mohamed bin Zayed University of Artificial Intelligence taking the lead. TII released the Arabic language model Noor in April 2022 and Falcon 7B and 40B model in March 2023. Inception released its Arabic-English bilingual Jais 13B in August. All models have performed well against global benchmarks, with TII's Falcon 40B and Falcon 180B both being ranked top by Hugging Face.
The undeniable achievement of Abu Dhabi's AI R&D community in developing world-class LLMs is a result of three main factors. Firstly, the UAE and Abu Dhabi as its capital has a clear vision for the role of AI in the future economy and the global leadership role that the country aspires to in defining the future of AI. Secondly, Abu Dhabi's huge oil wealth allows it to invest in capital intensive ventures and, when required, play the long game. Thirdly, Abu Dhabi understood early on that research and development required both the highest international standard of expertise and methodology, and a fresh approach to R&D management that would allow it to accelerate faster than other hubs. The LLM successes are proof positive that Abu Dhabi’s R&D strategy is succeeding.
IMO - It's a triumph for TII, barely more than three years after it was founded to have such a string of AI R&D successes, which sends a signal to the global R&D ecosystem that both TII as a lab, and Abu Dhabi is a hub, are to be taken seriously.
Commercially there are some rather obvious questions, such as how the Falcon series can create a sustainable ecosystem (of independent software vendors) and other developers or integrators).
After open-sourcing Falcon 40B in May as royalty free under an Apache 2.0 licence, with a tremendous response from developers, it has now withdrawn the licence. Meanwhile, Falcon 180B is simply not built to be an open source model.
So, it will be interesting to see when TII's roadmap for the Falcon LLM series emerges, in order that developers and enterprise users understand what to expect from TII if they commit to implementing and building on Falcon models.
Updated 7-Sep-23
LINKS
You may also like to read:
Will GenAI champion the Arabic language? (Middle East AI News)
R&D moves up Abu Dhabi's agenda (Middle East AI News)
MBZUAI releases first LLM library (Middle East AI News)
Can UAE-built Falcon rival global AI models? (Middle East AI News)