Meta’s Next Llama AI Models Are Training on Another ‘Bigger Than Anything’ GPU Cluster
Managing such a huge amount of chips to develop Llama 4 will likely pose unique technical challenges and require huge amounts of power. Meta CEO on Wednesday ignored analyst questions about limited access to energy in parts of the US that have hindered companies’ efforts to develop more powerful AI.
According to an estimatea cluster of 100,000 H100 chips would require 150 megawatts of power. The largest national laboratory supercomputer in the United States, El Capitanotherwise requires 30 megawatts of electricity. Meta expects to spend up to $40 billion in capital this year to equip data centers and other infrastructure, up more than 42% from 2023. The company expects that level of spending will grow even hotter next year.
Meta’s total operating expenses have increased about 9% this year. But overall revenue—largely from advertising—rose more than 22%, giving the company higher margins and bigger profits even as it poured billions of dollars into the Llama effort.
Meanwhile, OpenAI, considered the current leader in advanced AI development, is burning money despite charging developers for access to its models. What currently remains a non-profit venture said that they are training GPT-5, the successor to the model that currently supports ChatGPT. OpenAI has said that GPT-5 will be larger than its predecessor, but they haven’t said anything about the computer cluster it is using for training. OpenAI also said that in addition to scaling, GPT-5 will incorporate other improvements, including the one recently developed theoretical approach.
CEO Sam Altman said that GPT-5 will be “a significant leap forward” compared to its predecessor. Last week, Altman responded to a news report saying that OpenAI’s next frontier model will be released in December. by writing on X, “fake news is out of control.”
On Tuesday, Google CEO Sundar Pichai announced the company’s latest release Gemini family of innovative AI models is in development.
Meta’s open approach to AI is sometimes controversial. Some AI experts worry that making significantly more powerful AI models freely available could be dangerous because it could help criminals conduct cyberattacks or automate the design of weaponization study or biology. Although Llama was tweaked before release to limit misbehavior, removing these limitations is relatively simple.
Zuckerberg remains optimistic about the open source strategy, even as Google and OpenAI push for proprietary systems. “It seems pretty clear to me that open source is going to be the most cost-effective, customizable, reliable, performant, and easy-to-use option for developers,” he said. said Wednesday. “And I’m proud that Llama is leading the way on this issue.”
Zuckerberg added that the Llama 4’s new capabilities will be able to provide power a variety of features across Meta services. Today, the featured product based on the Llama model is a ChatGPT-like chatbot called Meta AI, available on Facebook, Instagram, WhatsApp and other apps.
Zuckerberg said more than 500 million people use Meta AI monthly. Over time, Meta hopes to generate revenue through advertising in this feature. “There will be an expanding set of queries that people use it for, and the monetization opportunity will exist over time as we get there,” Meta CFO Susan Li said on Wednesday’s call. With advertising revenue potential, Meta could subsidize Llama for others.
