Nvidia, Groq, and the 'Limestone Race' to Real-Time AI: Understanding the Shifting Paradigms of Compute Power and Enterprise Success
The article draws an analogy between the Great Pyramid's construction and technological growth, highlighting that progress isn't smooth but rather a series of sprints and plateaus, like massive limestone blocks. It revisits Moore's Law, noting the shift in compute power growth from CPUs, which plateaued, to GPUs, where Nvidia's CEO Jensen Huang strategically built a dominant position. The current wave of generative AI, driven by transformer architecture, also shows signs of paradigm shifts, similar to how GPUs took over from CPUs. An example cited is DeepSeek's ability to train a world-class model on a small budget using the Mixture-of-Experts (MoE) technique, a method also mentioned in Nvidia's Rubin press release concerning NVLink interconnect technology for acceleration.
The article uses the analogy of the Great Pyramid, which appears smooth from a distance but reveals massive, jagged limestone blocks up close, to illustrate the nature of technological growth. This growth is not a continuous, smooth incline but rather a series of 'staircases' or 'limestone blocks,' characterized by sprints followed by plateaus.
Historically, Gordon Moore's 1965 prediction of transistor count doubling annually, later revised by David House to compute power doubling every 18 months, held true for Intel's CPUs. However, CPU performance eventually 'flattened out like a block of limestone.' The article points out that while CPUs plateaued, the next 'limestone block' of compute growth emerged in GPUs. Nvidia's CEO, Jensen Huang, is credited with playing a 'long game,' building stepping stones through gaming, then computer vision, and more recently, generative AI, ultimately becoming a strong winner in this shift.
The 'illusion of smooth growth' extends to generative AI, which is currently driven by transformer architecture. Dario Amodei, Anthropic's President and co-founder, is quoted acknowledging the continued exponential growth, despite annual skepticism. However, the article suggests that just as CPUs plateaued and GPUs took the lead, there are signs that Large Language Model (LLM) growth is undergoing another paradigm shift. An example provided is DeepSeek's achievement in late 2024, training a world-class model on a remarkably small budget, partly by employing the Mixture-of-Experts (MoE) technique. This technique is also noted to have been mentioned in Nvidia's Rubin press release, specifically in relation to 'the latest generations of Nvidia NVLink interconnect technology... to accelerate' advancements.