Phi-2: Microsoft LLM

Microsoft Research has unveiled Phi-2, a 2.7 billion-parameter language model, as the latest addition to their Phi series. Despite its smaller size, Phi-2 competes with larger models like Mistral-7B and Llama-2 7B. Like its predecessor Phi-1.5, Phi-2 was trained on “textbook-quality” real and synthetic data. Phi-2 was built by embedding the knowledge of the 1.3B Phi-1.5 through a technique called scaled knowledge transfer.

At just 2.7B, Phi-2 outperforms 7B and 13B Mistral and Llama-2 models on composite benchmarks, and even beats the Llama-2-70B on complex reasoning tasks. What’s more, the model also shows improved handling of toxicity and bias, without explicit reinforcement learning from human feedback (RLHF). Phi-2’s success underscores the promise of training LLMs on textbook-quality data, and demonstrates the utility of high-quality synthetic data in LLM training pipelines.

• Efficiency: At only 2.7B, Phi-2 rivals 7B and 13B models across the board.
• Quality Training: Uses “textbook-quality” training data.
• Device Friendly: Runs efficiently on laptops and mobile devices
• Accessibility: Available for research on Azure AI Studio.