Top papers: CIFAR-10 94% in 3.29 Sec, Gemini Infinite Context Method, Microsoft’s Vasa-1

Achieve 94% accuracy on CIFAR-10 in just 3.29 seconds using a single NVIDIA A100 GPU, scaling up to 96% in 46.3 seconds with advanced techniques. Integrate strategies like patch-whitening, identity initialization, higher learning rate for biases, Lookahead optimization, multicrop TTA, and alternating flip for augmentation. Utilizing torch.compile for efficient GPU usage, this method significantly speeds up ML experiments and reduces costs, showing a 1.9× speed boost over previous records. Learn how these techniques can generalize across small-scale tasks and contribute to rapid model training.

94% on CIFAR-10 in 3.29 Seconds on a Single GPU

Achieves 94% accuracy on CIFAR-10 in 3.29 seconds on a single NVIDIA A100, scaling to 96% in 46.3 seconds. Integrates six techniques: patch-whitening and identity initialization, higher learning rate for biases, Lookahead optimization, multicrop TTA, and alternating flip for augmentation.

These strategies, combined with torch.compile for efficient GPU use, offer open-source scripts, markedly speeding up ML experiments and reducing costs, showing a 1.9× speed boost over previous records. Techniques generalize across small-scale tasks and contribute uniquely to the final speed, with alternating flip providing a notable 10% speedup.

“I think it’s nice to develop intuition for training neural nets with this thing. It can be used to test classic things like “when you double the batch size, is it really ideal to double the learning rate?” in < 1 minute”

Google Releases New Infinite Context Method

Can LLMs Handle Unlimited Context? Google researchers introduced a new concept called Infini-attention in their latest paper, enabling LLMs to process inputs of any length.

Comparison with Traditional Transformers: Typical transformers reset their attention memory after each context window to manage new data, losing previous context. For example, in a 500K token document split into 100K token windows, each segment starts fresh without memory from the others.

Infini-attention’s Approach: Infini-attention retains and compresses the attention memory from all previous segments. This means in the same 500K document, each 100K window maintains access to the full document’s context.

The model compresses and reuses key-value states across all segments, allowing it to pull relevant information from any part of the document.

How Infini-attention Works:

• Utilizes standard local attention mechanisms found in transformers.
• Integrates a global attention mechanism through a compression technique.
• Merges both local and global attention to manage extended contexts efficiently.
  In other words, the method effectively gives each window a view of the entire document, achieving what’s termed as “infinite context.”

Key Performance Metrics:

• 1B Model: Effectively manages sequences up to 1 million tokens.
• 8B Model: Achieves state-of-the-art results in tasks like summarizing books up to 500K tokens in length.

Key Highlights:

• Memory Efficiency: Constant memory footprint regardless of sequence length.
• Computational Efficiency: Reduces computational overhead compared to standard mechanisms.
• Scalability: Adapts to very long sequences without the need for retraining from scratch.

Given how important long-context LLMs are becoming having an effective memory system could unlock powerful reasoning, planning, continual adaption, and capabilities not seen before in LLMs. Great paper! Elvis Saravia

with Griffin and Infini-attention, it increasingly feels like Google leapfrogged Together and RWKV in the race for scaling up linear attention, and they shared a watercooler conversation with Anthropic or something. swyx

Sounds good on paper but scaling this to near-infinity would also mean inference would require crazy hardware resources, isn’t it?. Raúl Avilés Poblador

Microsoft’s Vasa-1: Hyper-Realistic Face Video And Lip-Sync From A Single Picture

Microsoft recently introduced VASA-1, an AI model that produces realistic talking face videos from a single static image and an audio clip.

The model outputs videos at 512×512 resolution and up to 40 frames per second (FPS), with a latency of only 170 milliseconds on NVIDIA RTX 4090 GPU systems.

Diffusion-Based Model Architecture

• Holistic Approach: Unlike traditional methods that may handle facial features separately, VASA-1 uses a diffusion-based model to generate holistic facial dynamics and head movements. This method considers all facial dynamics, including lip motion, expression, and eye movements, as parts of a single comprehensive model.
• Face Latent Space: The model works within a disentangled and expressive face latent space, enabling it to control and modify facial dynamics and head movements independently from other facial attributes like identity or static appearance.

Data-Driven and Disentangled Training

• Diverse Data: VASA-1 is trained on a large and diverse dataset, allowing it to handle a wide range of facial identities, expressions, and motion patterns. This training approach helps the model perform well even with input data that deviates from what it was trained on, such as non-standard audio inputs or artistic images.
• Disentanglement Techniques: The model’s training involves advanced disentanglement techniques, allowing for the separate manipulation of dynamic and static facial features. This is achieved through the use of distinct encoders for different attributes and a set of carefully designed loss functions to ensure effective separation of these features.

Extensive Benchmarks: VASA-1 has been rigorously tested against various benchmarks and has shown to significantly outperform existing methods in terms of realism, synchronization of audio-visual elements, and expressiveness of the generated animations.

The research acknowledges existing limitations, such as the model’s current inability to process full-body dynamics or to fully capture non-rigid elements like hair. Future work is planned to expand the model’s capabilities and address these areas.

Unfortunately this will assure that this technology gets a very brisk reactionary need to “act” by regulators almost like that was the point? Brian Roemmele

The eyes are always a dead giveaway, visible here as well. This is hands down the best demo yet. hou.mon