Grok 3 - 200k GPUs in Memphis and Somehow It Works

The Colossus Project

xAI crammed 200,000 H100 GPUs into Memphis and called it Colossus. For perspective, that's more GPUs than AWS, Google, or Microsoft allocate to any single customer. While normal companies spend 18-36 months building data centers (because permits, environmental studies, and not wanting to fuck up a billion-dollar project), xAI got theirs running in 122 days.

How did they pull this off? They cheated. Instead of building from scratch like proper data center companies, they grabbed a dead Electrolux factory from 2020. All the boring shit like structural engineering, power grid connections, and environmental permits were already sorted. They just had to gut it and stuff it with GPUs.

Memphis: Because They Need Stupid Amounts of Power

150MW continuous draw. That's not a typo - they're sucking down enough juice to power a small city. The Tennessee Valley Authority can actually deliver that much power without the grid shitting itself, which rules out pretty much everywhere else in America.

The power bill? They're not telling, but it's gotta be tens of millions per month just for electricity. Plus Memphis gets hot as balls in summer, so they're burning even more money keeping 200k H100s from melting into expensive paperweights.

Memphis also doesn't give a shit about environmental reviews like California does. Try building this in San Francisco - you'd still be waiting for permits in 2027.

Networking 200k GPUs: A Fucking Nightmare

3.6 Tbps per server. That's more bandwidth than most ISPs have for their entire customer base. xAI's specs aren't joking around - when you need to sync gradients across 200k GPUs, every nanosecond of network latency tanks your training performance. The NVIDIA networking architecture uses custom topology designed specifically for this scale.

Off-the-shelf networking gear would curl up in a corner and die. They had to build custom topology because Cisco's best switches become the bottleneck when you're trying to move model weights between 200k GPUs. High-performance computing networking at this scale requires specialized InfiniBand solutions. One misconfigured switch and your billion-dollar training run sits there doing nothing.

And shit breaks constantly. At this scale, you expect hundreds of GPUs to die every day. Network cards fail, switches overheat, cables get unplugged by accident. The software has to just deal with it and keep training around the failures.

Cooling 150MW: Don't Let the Magic Smoke Escape

Each H100 dumps 700W of heat when it's working hard. Multiply that by 200k and you're generating enough heat to warm a small town - around 140MW worth. That's on top of all the networking gear, storage, and other shit that also gets hot.

Air conditioning? Forget about it. You need liquid cooling that can handle the thermal output of a city. The cooling bill is probably 30% of their total operational costs. And if the cooling fails for even a few minutes, you just turned $50 billion worth of GPUs into very expensive e-waste. The Supermicro liquid cooling systems had to be specifically engineered for this deployment.

Memphis humidity doesn't help either. Sure, it's better than Arizona, but you're still fighting physics when you need to dump that much waste heat into the atmosphere.

What All This Hardware Actually Gets You

Grok 3 used 10x more compute than Grok 2 to train. That's not just throwing money at the problem - when you can dedicate 200k GPUs to a single training run, you can try model architectures that everyone else can only dream about.

The "Think" mode burns compute like crazy - maybe 50x more per query than normal inference. Most companies would go bankrupt running that on rented cloud GPUs. But when you own the hardware, you can afford to be wasteful with compute to get smarter answers.

OpenAI trains on maybe 25k GPUs max. Meta uses 30-50k. Google spreads their TPUs across a bunch of different projects. xAI can throw all 200k at one model and see what happens.

When Everything's Always Broken

At this scale, something dies every few minutes. GPUs overheat, network cards shit the bed, cooling pumps break, power supplies fry themselves. It's not "if" hardware fails, it's "how many things broke while I was getting coffee." The data center design specifications had to account for continuous hardware replacement cycles.

99% uptime means 2,000 GPUs are dead at any given moment. That's not a problem, that's Tuesday. Most data centers freak out when three servers fail. Here, losing a few hundred GPUs during a training run is just background noise.

The software has to be bulletproof. You can't pause a billion-dollar training run because GPU #47,293 decided to take a dirt nap. The training code expects failures and routes around them automatically.

The Million GPU Pipe Dream

They want 1 million GPUs eventually. That's completely insane. 750MW of power - half a nuclear plant's worth. The TVA power grid analysis would need to build new transmission lines just for them.

Cooling 750MW of waste heat? You'd need to divert a river. Memphis would become the GPU capital of America by accident, and the locals would probably hate them for it.

And the failures... if 200k GPUs means 2k are always broken, 1 million means 10k dead GPUs at any moment. You'd need an entire warehouse just for replacement parts and a small army of techs swapping hardware 24/7.

This is either visionary or completely fucking crazy. Probably both.

So they burned through stupid money building this thing. Grok 3 dropped in February 2025 and surprisingly, throwing 200,000 GPUs at the problem actually worked.

Andrej Karpathy (former OpenAI director, definitely not a hype guy) got early access and called it "around state of the art thinking model" - putting it up there with o1-pro, which OpenAI charges $200/month for.

The "Big Brain" Thing Is Real (And Expensive)

The "Big Brain" mode isn't marketing bullshit - it's what happens when you can afford to burn 50x more compute per query. Most AI models spit out the first thing that seems reasonable because compute time costs money. When you own 200k GPUs, you can afford to let it think harder.

It's brute force reasoning. Throw more compute at the problem until you get smarter answers. The reasoning traces actually show you how it's working through stuff, which is useful when you need to check if the AI is being an idiot.

Most models are like "trust me bro, here's your answer." This one shows its work like a high school math problem.

DeepSearch: Because They Can Afford To

DeepSearch is their research mode that runs multiple search strategies in parallel and synthesizes results. It's competing with Perplexity's DeepResearch but with more computational headroom to do deeper analysis.

This is only economically viable because they own the infrastructure. Running parallel research strategies on rented compute would cost a fortune per query. When you own the GPUs, you can afford to be wasteful with compute to get better results.

Early reports suggest it matches or beats existing research tools, which makes sense given the resource advantage. More compute doesn't always mean better AI, but for research and analysis tasks, it usually helps.

Real-Time X Integration (The Only Reason This Exists)

The main differentiator is native X platform integration. While other AI models are stuck with training data from months ago, Grok 3 can pull from X's real-time feed.

This requires processing millions of posts continuously to find relevant context for queries. The compute requirements are insane - you're basically running AI inference on the entire X firehose 24/7. Only viable when you own the hardware.

Most AI companies can't afford to process real-time social media at scale. It's computationally expensive and the value is questionable for general use cases. But if you own X anyway, might as well use the data.

Performance vs The Competition

xAI's benchmarks show Grok 3 beating GPT-4o, Gemini 2 Pro, Claude 3.5 Sonnet, and DeepSeek V3 on math, science, and coding tests. Take vendor benchmarks with salt, but Karpathy's independent assessment suggests it's legit.

The 10x compute advantage over Grok 2 training enables model architectures that competitors can't afford to attempt. When you can train on 200,000 GPUs instead of 20,000, you can experiment with approaches that would be financially impossible otherwise.

Pricing: Surprisingly Reasonable

Despite burning tens of millions monthly on infrastructure, access is surprisingly cheap. X Premium+ subscribers get it for $40/month. The xAI API is priced competitively with GPT-4 and Claude.

This only works because they're bleeding money to gain market share. Pure compute costs would make this completely unprofitable at current pricing. They're subsidizing it with X revenue and whatever other cash Musk can throw at it.

When you own 200k GPUs, you don't have to degrade performance when everyone else is competing for cloud instances. Most AI services slow down during peak usage because renting more compute costs money. xAI can just use more of their own hardware.

Context Windows and Memory

They haven't published exact context window sizes, but early reports suggest Grok 3 maintains coherence across very long documents and multi-session conversations. The Colossus infrastructure provides enough memory to support massive context retention without performance hits.

Most AI models struggle with long context because memory is expensive. When you have petabytes of high-speed memory available, you can afford to keep more context active. This enables applications that require understanding of very long documents or extended conversations.

The Technical Nightmare Nobody Talks About

Running inference on 200k GPUs simultaneously is an absolute clusterfuck from an engineering perspective. They had to write everything from scratch because Kubernetes would have a nervous breakdown, Docker Swarm would explode, and any normal load balancer would just give up.

3.6 Tbps per server. Most regional ISPs don't have that much total bandwidth. One fucked up switch config and your billion-dollar training run sits there doing nothing while you troubleshoot networking.

Memory management is brutal at this scale. H100s are finicky about data layouts - organize your tensors wrong and memory bandwidth goes to shit. You need custom NUMA awareness, GPU memory pools that don't fragment under load, and cache invalidation strategies that don't cascade through the entire cluster.

The NVIDIA drivers probably weren't tested at this scale. Nobody runs 200k GPUs. They're definitely hitting edge cases that would make a normal sysadmin weep.

What This Means for Everyone Else

xAI is proving that infrastructure is the new moat. While other companies rent compute and optimize for cost efficiency, xAI built dedicated infrastructure optimized for capability.

This creates a feedback loop: better infrastructure enables more capable models, which justify larger infrastructure investments, which enable even better models. Other AI companies are stuck optimizing for rental costs while xAI optimizes for performance.

The roadmap to 1 million GPUs suggests this is just the beginning. If they can scale efficiently, the capability gap will only grow. Other companies will need similar infrastructure investments to stay competitive.

The question is whether throwing more compute at the problem continues to yield better results, or if there are diminishing returns. So far, the scaling laws suggest more compute = better AI, but physics and economics will eventually set limits.