The version of Google’s Gemini you can use right now doesn’t beat OpenAI’s GPT-4

The AI arms race is heating up between Google CEO Sundar Pichai and OpenAI CEO Sam Altman.

Getty Images

Google’s been hyping up Gemini as the AI that can outperform industry leader OpenAI’s GPT-4.However, that version of Gemini won’t be available to users until next year.Instead, a version that’s surpassed by GPT-4 is available on Google’s Bard chatbot.

Google has been under pressure to counter the idea that it’s fallen behind rivals in the AI race.

Since May, the company has been hyping up the capabilities of its new AI model, Gemini, and its ability to outperform industry leader OpenAI’s GPT-4.

However, Google’s buzzy launch of Gemini on Wednesday doesn’t quite meet that goal just yet.

In the blog post announcing Gemini, Google DeepMind CEO Demis Hassabis shared that Gemini outperforms OpenAI’s GPT-4 (the latest model available to paying users of ChatGPT) on a slew of measures.

There’s a catch, though, because Google said that Gemini would come in three versions: Ultra, Pro, and Nano. The performance touted by Hassabis only refers to Gemini Ultra – which won’t be available until next year.

The version available now is Gemini Pro, which can be accessed through Google’s Bard chatbot, per the blog post.

Google’s technical paper released alongside Gemini’s announcement revealed that Gemini Pro could outperform GPT-3.5 on most measures – but failed to beat OpenAI’s GPT-4.

The only version of Gemini that outperforms GPT-4 isn’t yet available to users.

Google

In the blog post, Hassabis wrote that Ultra would be available to users through “Bard Advanced” early next year.

The Information initially reported on Sunday that Google was planning to delay the release of Gemini until 2024 over difficulties with handling prompts in non-English languages.

“Now, we’re taking the next step on our journey with Gemini, our most capable and general model yet, with state-of-the-art performance across many leading benchmarks,” wrote Google CEO Sundar Pichai in the company’s Gemini announcement.

However, the launch has had a mixed reception online.

“The big deal is that it appears to be the first model to beat GPT-4,” wrote Ethan Mollick, a professor at the University of Pennsylvania’s Wharton School, on X. “The fascinating thing is that it does it by just a tiny bit.”

Yacine Jernite, a researcher at AI company Hugging Face, wrote on X: “60-page report, 2 paragraphs on data amounting to ‘trust us we did it well.’ Do better.”

Meanwhile, Jesse Dodge, a research scientist at the Allen Institute for AI, wrote that despite Google’s claims that its training data for Gemini was key to its performance, the company provided “almost no information about how it was made, how it was filtered, or its contents.”