At the highly anticipated Google I/O 2023 event, the tech giant introduced its latest groundbreaking advancement in the field of artificial intelligence: PaLM 2. Serving as the foundation for several upcoming Google products, including Google Generative AI Search, Duet AI in Google Docs and Gmail, and Google Bard, PaLM 2 is a remarkable general-purpose large language model. This article aims to provide an in-depth overview of the PaLM 2 AI model, exploring its features, capabilities, and how it compares to its predecessor, GPT-4.
What is Google’s PaLM 2 AI Model?
PaLM 2 represents the most recent release of Google’s Large Language Model (LLM). This highly capable model excels in advanced reasoning, coding, and mathematics, making it a versatile tool. Supporting over 100 languages, PaLM 2 is a successor to the earlier Pathways Language Model (PaLM) that made its debut in 2022. Compared to its predecessor, PaLM 2 has undergone significant advancements. While the original PaLM was trained on an impressive 540 billion parameters, the newer PaLM 2 model is smaller in size yet faster and more efficient.
Understanding the PaLM 2 AI Model
Although Google has not explicitly disclosed the parameter size of PaLM 2 in its 92-page technical report, reports from TechCrunch suggest that one of the PaLM 2 models has been trained on 14.7 billion parameters. This parameter count is considerably lower than that of PaLM 1 and other competing models. Speculation on Twitter by researchers suggests that the largest PaLM 2 model could be trained on around 100 billion parameters, still significantly smaller than the parameter count of GPT-4.
For comparison, OpenAI’s GPT-4 model is reportedly trained on a staggering 1 trillion parameters, showcasing the magnitude of its scale. GPT-4 is approximately ten times larger than PaLM 2 in terms of parameters.
The Smaller Footprint of PaLM 2
Google emphasizes that size isn’t always an indication of superiority. In its official blog post, the company highlights the importance of research creativity in creating exceptional models. While the specific research creativity employed in PaLM 2 remains undisclosed, it is likely that Google has utilized techniques such as Reinforcement Learning from Human Feedback (RLHF), compute-optimal scaling, LoRA (Low-Rank Adaptation), instruction tuning, and high-quality datasets. These techniques contribute to achieving impressive results with a relatively smaller model.
Highlight Features of PaLM 2
PaLM 2 boasts a range of features that set it apart. Firstly, the model excels in common sense reasoning, with Google affirming that PaLM 2’s reasoning capabilities rival those of GPT-4. In various reasoning tests such as the WinoGrande commonsense test, PaLM 2 achieved a score of 90.2, outperforming GPT-4’s score of 87.5. In other tests like the ARC-C test, GPT-4 achieved a slightly higher score of 96.3 compared to PaLM 2’s 95.1. PaLM 2 also outperforms GPT-4 in reasoning evaluations including DROP, StrategyQA, and CSQA.
Additionally, PaLM 2’s multilingual capabilities enable it to understand idioms, poems, nuanced texts, and riddles in different languages. It surpasses literal word meanings and comprehends the ambiguous and figurative aspects of language. Pre-training on parallel multilingual texts and a corpus of high-quality multilingual data enhance PaLM