DeepSeek's new chatbot boasts an impressive introduction: "Hi, I was created so you can ask anything and get an answer that might even surprise you." This AI, a product of the Chinese startup DeepSeek, has rapidly become a major market player, even contributing to a significant drop in NVIDIA's stock price. Its success stems from a unique architecture and training methodology, incorporating several innovative technologies.
Multi-token Prediction (MTP): Unlike traditional word-by-word prediction, MTP forecasts multiple words simultaneously, analyzing sentence segments for enhanced accuracy and efficiency.
Mixture of Experts (MoE): This architecture leverages multiple neural networks to process input data, accelerating AI training and boosting performance. DeepSeek V3 utilizes 256 networks, activating eight for each token.
Multi-head Latent Attention (MLA): This mechanism focuses on crucial sentence elements. MLA repeatedly extracts key details, minimizing the risk of overlooking important information and enhancing nuanced understanding.
DeepSeek initially claimed a remarkably low training cost of $6 million for its powerful DeepSeek V3 model, using only 2048 GPUs. However, SemiAnalysis revealed a far larger infrastructure: approximately 50,000 Nvidia Hopper GPUs (including 10,000 H800s, 10,000 H100s, and additional H20s) spread across multiple data centers. This represents a total server investment of roughly $1.6 billion, with operational expenses estimated at $944 million.
DeepSeek, a subsidiary of the High-Flyer hedge fund, owns its data centers, providing complete control over optimization and faster innovation implementation. This self-funded approach enhances flexibility and decision-making speed. Furthermore, the company attracts top talent, with some researchers earning over $1.3 million annually, primarily from Chinese universities.
The $6 million figure, therefore, appears to be a significant understatement, representing only pre-training GPU costs. The actual investment in AI development exceeds $500 million. Despite this, DeepSeek's streamlined structure allows for efficient innovation implementation compared to larger, more bureaucratic companies.
DeepSeek's success showcases the potential of a well-funded independent AI company to compete with industry giants. While the "revolutionary budget" claim is arguably exaggerated, the company's success is undeniable, fueled by substantial investment, technological breakthroughs, and a highly skilled team. The contrast is striking when considering competitor costs; DeepSeek's R1 model cost $5 million, while ChatGPT4 cost $100 million. Even with the clarified costs, DeepSeek remains significantly cheaper than its competitors.
