When OpenAI or Anthropic ship a new model, the launch page tells you it's stronger at everything. Benchmark charts, a few cherry-picked demos, a number that went up. None of it tells you whether it's better at your work, and your work is the only benchmark that matters.
If you came from the video, that's the idea: don't trust the launch, test it. Here's the part that doesn't fit in sixty seconds: the three tests I run, the prompt that makes the comparison fair, the scorecard I fill in after, and the traps that make a test look decisive when it isn't.
The three tests
A benchmark grades the model on problems someone else chose. My question is narrower: is it better at the thing I do over and over, on my inputs, than the model I already pay for? So I don't test in the abstract. I run the new model head to head against my current one, on the same inputs, and keep whichever wins. If nothing wins, I keep what I have. I run these three in order: the first two check whether it can do a real job at all, the third decides whether I switch.
Give it something big
Hand it a whole project, not a one-liner. "Here are a thousand customer reviews. Find the five biggest complaints, then draft the offer, the sales page, and a launch plan around them." Give it your real files, your real numbers, and what "done" looks like. You're testing whether it can carry what it learns in step one into step four, instead of forgetting the reviews by the time it writes the plan.
Give it something you do every week
Pick a task you repeat. For me it's content research: here are the accounts to watch, three posts that did well and why, now find this week's strongest angles and who they're for. Run it once, fix what it misses, then imagine running it every Monday. You're testing how much it takes off your plate on a normal week, not on a demo. Brilliant once but needy every time isn't saving you anything.
Make both compete on something you're stuck on
This is the one that decides it. Take a real problem you haven't cracked, getting traffic but no sales, say, and hand both models the same page, the same numbers, and the same question: what would you change, and why. Same prompt, no hints. Then check both answers against what you know is true. Which found something you'd act on, and which needed more corrections to get there? That gap is your answer.
Make the comparison fair before you trust it
Most comparisons break before they start because the two models got different prompts, different context, or a grade colored by which one you were rooting for. Fix it with one habit: write the prompt once, paste the identical thing into both, change nothing but the model. Here's the wrapper I put around whatever the real task is, so both answer the same question in the same shape.
The task: [the exact job, in one or two sentences]
Here is everything you get to work from:
[paste the same inputs into both: files, numbers, examples, my voice]
What I need back:
- [the deliverable, in the exact format I'd actually use]
- Cite which input each claim comes from.
- If something is missing or unclear, say so instead of guessing.
Do not ask me follow-up questions. Give me your best answer with
what you have, and note where you were unsure.Two rules make the result trustworthy. Reset the chat between runs so the second model can't see the first's answer. And grade blind: label the outputs A and B, judge them, then reveal which is which. It's uncomfortable how often the one you were rooting for is B.
A worked example: a newsletter writer testing a new model
Say you write a paid newsletter and already pay for one model to draft it. A new one launches, the timeline says it's a better writer, and you want to know if it's worth switching before rebuilding your workflow around it.
The big job is a full issue. You hand both models the same brief: this week's topic, links to three past issues so they learn your voice, your usual structure, and "draft the whole issue, subject line to sign-off." You're watching whether each holds your voice to the end or drifts into generic newsletter mush by the third section.
The weekly task is the subject line. Give both the same drafted issue and ask for ten subject lines plus one sentence on why each would make a subscriber open. You'd run this every week, so the question isn't "can it write a clever line once," it's "would I ship any of these without rewriting them."
The stuck problem is your open rate, which has slid for a month. You give both the same six issues, the same open-rate numbers, and ask what's changing and what to test next. One says "post consistently and write better subject lines," which is a horoscope. The other notices your opens drop on issues where the subject line promised a how-to and the body was an essay, and names the pattern. That answer is worth paying for. The first isn't.
Then you fill in the scorecard, once per test, and the winner is usually obvious by the time the grid is full.
Task tested: [big job / weekly task / stuck problem]
| Current model | New model
------------------------------------------------------
Held my voice/format? | |
Useful, specific finding?| |
Errors or made-up facts | |
Corrections I had to make| |
Time to a usable result | |
Cost of the run | |
Winner for THIS task: [which one, and why in one line]
Would I switch my workflow? [yes / no / not yet]What goes wrong, and how to catch it
The most common trap is grading on charm. A model that writes with confidence reads as smarter even when it's more wrong. Check the claims against your real numbers before you score anything. In the newsletter example, the model that "sounded" like a strategist was the one giving generic advice, and you only caught it because you looked at the actual open rates.
The second trap is the lucky run. You test once, it nails it, you switch, and then it's mediocre for a week. One good answer is a coincidence, not a verdict. Run the weekly test three or four times on different inputs, because the point of that test is consistency, not a highlight reel.
The third is ignoring cost and speed. A model with a slightly better answer that costs three times as much or takes four times as long can still be the wrong choice for a task you run daily. Keep cost and time on the scorecard so a marginal quality win doesn't quietly blow up your bill. If you're weighing that across your whole stack, my guide on how to choose which AI tool to pay for walks through it, and which AI tools are worth paying for covers the bigger spending question these tests feed into.
The last is switching for the wrong reason. New is exciting, and it's easy to move because a model is newer rather than better at your job. If the scorecard says tie, stay. Relearning a tool and rebuilding your prompts costs real time, and a tie doesn't cover it. If a test keeps failing in the same place, the problem may be the workflow around the model, not the model, which is what finding out where your AI workflow breaks digs into. Rerun the tests when the next version ships.
A launch page is a claim, not a verdict. Run the same job through the model you pay for now and the new one, grade the results against your real numbers, and keep whichever earns it. Most of the time the honest answer is "not yet," and that's a win too.