Jerome Pesenti has some reasons to celebrate Meta’s decision last week to release Llama 3, a powerful open source large language model that anyone can download, run, and develop.
Pesenti used to be vice president of artificial intelligence at Meta and says he often pressured the company to consider releasing its technology for others to use and develop. But his main reason to rejoice is that his new startup will have access to an AI model that he says is very close in power to OpenAI’s industry-leading GPT-4 text generator, but considerably cheaper to run and more. open to external scrutiny and modification. .
“Last Friday’s release really seems like a game changer,” Pesenti says. His new company Sizzle, an AI tutor, currently uses GPT-4 and other AI models, both closed and open, to build problem sets and lesson plans for students. Its engineers are evaluating whether Llama 3 could replace the OpenAI model in many cases.
Sizzle’s story may portend a broader shift in the balance of power in AI. OpenAI changed the world with ChatGPT, unleashing a wave of investment in AI and attracting more than 2 million developers to its cloud APIs. But if open source models prove competitive, developers and entrepreneurs may decide to stop paying to access the latest model from OpenAI or Google and use Llama 3 or one of the other increasingly powerful open source models that are emerging.
“It’s going to be an interesting horse race,” Pesenti says of the competition between open models like Llama 3 and closed ones like GPT-4 and Google’s Gemini.
Meta’s previous model, Llama 2, was already influential, but the company says it made the latest version more powerful by feeding it larger amounts of higher-quality training data, with new techniques developed to filter out redundant or confusing content and select the better. combination of data sets to be used.
Pesenti says that running Llama 3 on a cloud platform like Fireworks.ai It costs only one-twentieth the cost of accessing GPT-4 via an API. He adds that Llama 3 can be configured to respond to queries extremely quickly, a key consideration for developers at companies like his that rely on models from different vendors. “It’s an equation between latency, cost and accuracy,” she says.
Open models seem to be falling at an impressive rate. A couple of weeks ago, I walked into startup Databricks to witness the final stages of an effort to build DBRX, a language model created that was briefly the best open one out there. That crown now belongs to Llama 3. Ali Ghodsi, CEO of Databricks, also describes Llama 3 as “revolutionary” and says the larger model “is approaching the quality of GPT 4, which levels the playing field between Open and Closed Source LLM.”
Llama 3 also shows the potential of making AI models smaller, so they can run on less powerful hardware. Meta released two versions of its latest model, one with 70 billion parameters (a measure of the variables it uses to learn from training data) and another with 8 billion. The smaller model is compact enough to run on a laptop, but it’s remarkably capable, at least in WIRED’s tests.
Two days before the launch of Meta, Mistrala French artificial intelligence company founded by alumni of the Pesenti team in Meta, open source Mixtral 8x22B. It has 141 billion parameters but uses only 39 billion of them at a time, a design known as expert mixing. Thanks to this trick, the model is considerably more capable than some much larger models.
Meta is not the only tech giant releasing open source AI. This week Microsoft launched Phi-3-mini and Apple launched OpenELMtwo small but capable free-to-use language models that can run on a smartphone.
The coming months will show whether Llama 3 and other open models can truly displace premium AI models like GPT-4 for some developers. And even more powerful open source AI is coming. The company is working on a massive, 400 billion-parameter version of Llama 3 that chief AI scientist Yann LeCun says should be one of the most capable in the world.
Of course, all this openness is not purely altruistic. Meta CEO Mark Zuckerberg says he is opening up his AI models Ultimately, it should benefit the company. reducing the cost of the technologies on which it depends, for example by generating compatible tools and services that Meta can use itself. He didn’t say that it might also be beneficial for Meta to prevent OpenAI, Microsoft or Google from dominating the field.