Deepdive: Many-Shot learning, AutoCrawler, Megalodon
Large language models (LLMs) are revolutionizing various fields, but their capabilities can be limited. This is where exciting new research comes in! By tackling challenges like limited learning from examples and information processing constraints, these advancements are making LLMs more adaptable, efficient, and powerful. Let’s dive into three key breakthroughs: Many-Shot In-Context Learning, AutoCrawler, and Megalodon.
Many-Shot In-Context Learning
Problem: Large language models are limited by few-shot in-context learning (ICL), which restricts adaptability and performance in complex tasks.
Solution: The research expands ICL to many-shot scenarios using larger context windows and hundreds of examples. It introduces Reinforced ICL with model-generated rationales and Unsupervised ICL that eliminates rationales entirely.
Results: Many-shot ICL significantly improves task performance, showing gains in adaptability and bias mitigation. It enhances reasoning and complex problem-solving, effectively learning high-dimensional functions.
AutoCrawler: A Progressive Understanding Web Agent
Problem: Traditional web crawlers struggle with adaptability and scalability in new environments, while generative agents based on large language models lack performance and reusability in open-world scenarios.
Solution: AutoCrawler, a two-stage framework that combines LLMs with crawlers, uses a progressive understanding approach leveraging the hierarchical structure of HTML. It includes top-down and step-back operations to refine actions and prune irrelevant HTML, enhancing efficiency.
Results: AutoCrawler significantly outperforms the state-of-the-art baseline in crawler generation tasks. Comprehensive experiments demonstrate its effectiveness in generating stable and executable action sequences for diverse and changing web environments.
Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length
Problem: Transformers face scalability issues with long sequences due to quadratic complexity and weak length extrapolation, while alternative models like linear attention underperform in pretraining efficiency and accuracy.
Solution: Megalodon introduces an architecture with unlimited context length, utilizing components like complex exponential moving average (CEMA) and normalized attention for enhanced efficiency and capability.
Results: In comparison with Llama2, Megalodon demonstrates superior efficiency at a scale of 7 billion parameters and 2 trillion training tokens, achieving a training loss of 1.70, which positions it between the performance benchmarks of Llama2’s 7B and 13B models.
