The multi-model era just started. Stop building for one vendor
Gemini's launch means the frontier is no longer one company. What benchmark season teaches enterprises about evaluating AI — and what it can't tell you.
🧭 The multi-model era just started. Stop building for one vendor.
Last week Google launched Gemini — three sizes, natively multimodal, with Ultra claiming to be the first model to outperform human experts on MMLU and benchmark wins over GPT-4 on a long list of tasks. Within days came the counter-discourse: questions about how the launch demo was edited, and arguments over which benchmark comparisons were fair.
For enterprise leaders, I'd suggest the specific scores are the least important part of the story. Two other things matter more.
First: the frontier is now officially plural. For a year, "AI strategy" in many organisations quietly meant "our OpenAI relationship, via our cloud contract." That was defensible when one lab was clearly ahead. It is not a strategy anymore — it is a dependency. With serious frontier models now coming from multiple labs, and clearly more ahead, the architecture question changes: can you evaluate a new model against your workload in weeks, and swap it in without a rewrite? If the answer is no, every future launch week is a reminder of your switching costs, not your options.
Second: benchmark season is a lesson in what evaluation is for. MMLU tells you a model can answer graduate-level exam questions. It tells you nothing about whether it can draft a compliant proposal from your templates, answer policy questions from your documents without inventing clauses, or handle your customers' phrasing. The only benchmark that matters is one built from your own work: a fixed set of real tasks, with known good answers, run against every candidate model under identical conditions. It is unglamorous, it takes a few weeks to build once — and it converts every future launch from a hype cycle into a data point.
The organisations that internalise both points get a compounding advantage: they greet each new model with a bake-off instead of a debate, and they negotiate with every vendor from a position of credible mobility.
A concrete exercise before year-end planning closes: take your top three AI use cases and write down, for each, the twenty tasks that would constitute a passing grade. If your team cannot produce that list, the gap in your AI strategy is not model access — it is that you have not yet defined what "good" means in your own terms. No launch event will answer that for you.
Reading: https://www.cnbc.com/2023/12/06/google-launches-its-largest-and-most-capable-ai-model-gemini.html