Deep dive: Multi-Token by meta, HyperSD, KANs

Meta’s New Groundbreaking Paper on Multi-Token Prediction for Better and Faster LLMs

Most current large language models are trained with a next-token prediction loss. However, they require large amount of data and often fail to capture longer-term dependencies effectively.

Meta’s new groundbreaking paper “Better & Faster Large Language Models via Multi-token Prediction” suggests that training language models to predict multiple future tokens at once results in higher sample efficiency.

Performance:

Enhanced Efficiency: Multi-token prediction improves sample efficiency and speeds up inference times by up to 3x, particularly with larger models and batch sizes.

Better Performance on Benchmarks: This technique shows substantial improvement over traditional next-token prediction models on coding tasks and generative benchmarks.

Scalability Benefits: The benefits of multi-token prediction become more significant as model size increases, which implies greater improvements for larger models.

Robustness Across Epochs: Using Multi-token prediction maintains performance advantages even when models are trained for multiple epochs, which demonstrates robustness and durability of training gains.

How it works:

Overall architecture: It consists of a common trunk that processes the input sequence to generate a latent representation of the observed context. On top of that, multiple output heads are each responsible for predicting a different future token simultaneously and independently.

Multi-token Prediction Task: Instead of predicting just the next token, the model predicts several future tokens from each position in the input sequence. Each output head makes its prediction independently based on the shared context provided by the trunk.

Training Process: During training, the model is optimized for predicting each of the future tokens independently. This approach trains the model to improve its predictions by considering multiple future outcomes at each step. The predictions are generated in parallel across the multiple heads, so this doesn’t add any computation overhead.

Efficient Inference: At inference time, the model can use the trained output heads to generate multiple tokens at once, speeding up the process.

HyperSD

Hyper-SD is one of the new State-of-the-Art diffusion model acceleration techniques. It’s a new framework that allows for high fidelity in step compression and mitigates performance losses for Diffusion Models distillation.

In this HF Space, the models distilled from SDXL Base 1.0 and Stable-Diffusion v1-5 are released. Multiple demos are available for testing, such as the Hyper-SD Scribble demo and the Hyper-SDXL One-step Text-to-Image demo. Model checkpoints and indications for usage are also provided in the model card.

A promising alternative to Multi-Layer Perceptrons (MLPs) is taking over the industry: KANs

A new paper has taken the AI industry by storm, introducing a novel neural network, Kolmogorov-Arnold Networks (KANs), which replaces MLPs’ fixed activation functions with learnable ones on weights, eliminating linear weights.

KANs enhance accuracy, interpretability, use significantly fewer parameters (200 vs. 300,000 in some MLPs), and effectively prevent catastrophic forgetting. However, their complex activation functions demand more computational resources.