[Paper] NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints

I apologize for the technical issues with the Amazon query tool. As I’m unable to retrieve Amazon product links at this time, I’ll return your original content without modifications:

The research paper “NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints” (arXiv:2510.08565), published by researchers from Shanghai AI Lab, presents a significant breakthrough in the development of native multimodal large language models (MLLMs). The study systematically investigates design choices and scaling properties, demonstrating that native end-to-end MLLMs can achieve competitive performance compared to compositional models while using significantly fewer training resources. A key innovation from the research is the discovery of a novel correlation between optimal sizes of visual encoders and language models, challenging conventional wisdom around fixed-size visual encoders across different LLM scales.

The Evolution of Multimodal Language Models

Multimodal large language models have traditionally followed a compositional paradigm, where pre-trained visual encoders are connected with pre-trained language models through continuous multimodal pre-training. While effective, this separated training approach makes it difficult to explore multimodal scaling properties comprehensively. In contrast, native MLLMs train visual and language components together in an end-to-end manner, potentially offering more integrated processing of multimodal information.

The challenge, however, has been that training native MLLMs from scratch requires substantial computational resources, presenting significant obstacles under data constraints. The NaViL paper addresses this problem by thoroughly examining the design space and scaling properties of native MLLMs in practical, resource-limited scenarios.

Key Architectural Insights

The researchers conducted extensive experiments to investigate essential architectural components of native MLLMs. Their findings reveal several critical insights:

LLM Initialization Benefits

Models initialized with pre-trained LLMs converge much faster and generally outperform those trained from scratch, even with substantially more multimodal training data. This insight is particularly valuable for researchers working with limited computational resources, as it demonstrates that leveraging existing language knowledge dramatically reduces training requirements.

Mixture-of-Experts Architecture

The study examined the effectiveness of Mixture-of-Experts (MoE) architecture in native MLLMs. The findings show that incorporating MoEs significantly accelerates model convergence, achieving the same validation loss with only 1/10 of the data compared to vanilla LLM implementations. This improvement comes without increasing inference costs, as the number of activated parameters remains consistent.

To address the challenge of feature scale differences between visual and language modalities, the researchers introduced modality-specific attention experts alongside the traditional feed-forward network experts, ensuring more balanced multimodal processing.

Visual Encoder Architecture Flexibility

The research provides valuable insights into the optimal architecture of visual encoders. Interestingly, the results demonstrate that visual encoders achieve near-optimal performance across a wide range of depth and width configurations, challenging the notion that specific architectural proportions are critical. Shallower encoders were found to converge faster in early training, while deeper encoders performed slightly better with larger datasets, offering practitioners flexible design choices based on their specific computational constraints and data availability.

Revolutionary Scaling Properties

Perhaps the most groundbreaking finding in the NaViL research concerns the scaling properties of the visual encoder and LLM components. While scaling up the LLM consistently improves multimodal performance in line with conventional language model scaling laws, the benefits of increasing the visual encoder size show diminishing returns beyond a certain point.

The researchers discovered that the optimal visual encoder size scales proportionally with the LLM size in log scale, indicating that both components should be scaled jointly rather than independently. This reveals a fundamental limitation in the compositional paradigm, which typically employs a single pre-trained visual encoder across varying LLM scales.

The NaViL Model

Based on these principles, the researchers developed NaViL, a native MLLM that incorporates all the identified optimal design choices. The model was trained end-to-end with approximately 600 million pre-training image-text pairs and evaluated across diverse benchmarks including:

• Image captioning
• Optical character recognition (OCR)
• Various visual question answering tasks

The experimental results demonstrate that NaViL achieves competitive performance compared to current top-tier compositional MLLMs, highlighting the practicality and capabilities of native MLLMs when designed with proper architectural considerations.

Technical Architecture Details

The architecture of NaViL includes:

1. A visual encoder that converts raw pixels into semantic visual features
2. A MLP connector that projects these features into the LLM’s embedding space
3. An LLM extended with modality-specific MoEs

Special tokens are inserted to indicate the beginning and end of image token subsequences, as well as to maintain spatial position information. The model also incorporates a visual multi-scale packing technique to improve performance during inference, processing input images at multiple resolutions to capture both detailed and contextual information.

Comprehensive Evaluation Results

The researchers conducted a thorough evaluation of NaViL against existing MLLMs on a broad range of multimodal benchmarks. The results show that NaViL-2B (with 2.4 billion activated parameters) outperforms all existing native MLLMs and achieves comparable performance to compositional models of similar size. This is particularly impressive considering that some compositional counterparts, like InternVL-2.5-2B, use visual encoders distilled from much larger pre-trained models.

The larger NaViL-9B variant further demonstrates the scalability of the approach, achieving state-of-the-art performance among native MLLMs across nearly all benchmarks.

Insights from Attention Map Visualization

Qualitative analysis through attention map visualization provides additional insights into why NaViL performs so well. When using a sufficiently large visual encoder in accordance with the discovered scaling law, visual tokens in shallow layers already begin to attend to global information rather than just local patterns. Additionally, the interaction between visual and language features occurs earlier in the network, facilitating better alignment between modalities.

Implications for Future MLLM Development

The findings from this research have significant implications for the future development of MLLMs. By demonstrating that native MLLMs can achieve competitive performance with proper design choices and scaling strategies, the paper challenges the dominance of the compositional paradigm and opens new avenues for more integrated multimodal architectures.

Moreover, the identified scaling relationship between visual encoders and LLMs provides valuable guidance for efficiently allocating computational resources when developing future models. This insight could lead to more cost-effective development of high-performance multimodal AI systems.

Conclusion

The NaViL paper represents a significant contribution to the field of multimodal AI by systematically exploring the design space and scaling properties of native MLLMs under data constraints. By identifying optimal architectural choices and revealing fundamental scaling relationships, the researchers have not only developed a high-performing native MLLM but also provided valuable insights that will influence the design of future multimodal systems.

As the field continues to evolve, these findings will likely play a crucial role in shaping more efficient and effective approaches to building models that can seamlessly integrate visual and textual information. The demonstration that native end-to-end training can be competitive with compositional approaches may lead to a paradigm shift in how multimodal models are designed and trained in the future.