How to Choose an AI Model: A Plain-English Guide

By Daily Tech Wizard

How to Choose an AI Model (The Short Version)

Define the job first, in one sentence. Then decide what a wrong answer costs you - a slightly dull email, or a client complaint.

Pick a starting strategy: cheapest first if the job is high-volume and low-stakes, most capable first if mistakes are expensive. Write down five real tasks you genuinely do, run the identical prompt through two or three candidates, and score the results yourself in a spreadsheet.

Re-test when your needs change or when something new lands. That's it. The method is the skill - not the name of whatever model is winning this month.

Why There Is No Single 'Best' AI Model

This guide will not name a best model. Not out of cowardice - out of maths.

Model line-ups change faster than articles get updated. Any page that crowns a winner is a page that quietly goes stale, and you end up making a decision on evidence that expired.

But there's a deeper problem. "Best" is an output of whichever benchmark you happened to read.

Look at how serious evaluation work is structured. Stanford's Center for Research on Foundation Models runs HELM, and it has deliberately split into separate leaderboards for separate purposes rather than one grand ranking.

There's HELM Safety, which pulls together five safety benchmarks across six risk categories - violence, fraud, discrimination, sexual content, harassment and deception. And there's MedHELM, built on a taxonomy of five categories, 22 subcategories and 121 clinical tasks drawn from 35 benchmarks.

Different benchmark, different winner. A model that tops a safety board may sit mid-table on clinical tasks, and vice versa.

Worth knowing too: HELM is built for transparent, reproducible evaluation - you can inspect the requests and responses sent to models, and reproduce results through the open-source HELM framework. That's exactly why it's a better reference point than a screenshot of a leaderboard somebody posted to social media. See the HELM capabilities leaderboard for how that transparency is presented.

So the honest answer to "which large language model is best?" is: best at what, measured how, and does that match the thing you're trying to do?

The 6 Questions That Decide Your Model

Here's the trick most people get backwards. They start comparing, get dazzled by one demo, and then reverse-engineer their criteria to justify the thing that impressed them.

Anthropic's own documentation on choosing a model boils the decision down to three things - capabilities, speed and cost - and recommends you establish your criteria before you start comparing. Sensible advice, and it applies just as much to picking a chat subscription as to picking a model inside an API.

Microsoft's Azure Architecture Center goes further and frames model selection as a decision tree, filtering candidates on task fit, cost, context window size, security and compliance, region availability, deployment strategy, domain specificity, performance and tunability.

That's written for architects. Below is the same logic, translated into six questions a normal person can answer over a cup of tea.

1. What Is the Job? (Task Fit)

Write the job down in one sentence, with a verb in it. Not "I need AI for my business" - that's not a job. "I need to turn rough voice notes into polite customer emails" is a job.

Task fit is the first filter for a reason. A model tuned for long reasoning chains is overkill for reformatting addresses, and a fast chat model may be the wrong tool for reading a 60-page contract.

If you can't write the sentence, you're not ready to compare. You're still shopping.

2. How Much Does a Wrong Answer Cost You?

This is the question almost nobody asks, and it decides more than any benchmark.

Drafting a first pass at a birthday speech? A wrong answer costs you thirty seconds. Quoting a job, summarising a client's medical history, or calculating something you'll put on an invoice? Different story entirely.

Hallucination - the polite industry word for confidently made-up detail - is a feature of how these systems work, not a bug you can shop your way out of. What changes with a better model is the rate, not the existence.

So grade your own job honestly. Low stakes, high volume, you check nothing? Go cheap. High stakes, low volume, somebody relies on it? Go capable, and check it anyway.

3. Cheapest First or Most Capable First?

Anthropic's documentation describes two sensible starting strategies, and both are legitimate.

Efficiency-first - start with a faster, cheaper model and move up only if it falls short.
Capability-first - start with the most capable model, improve your prompts until the output is right, then evaluate whether you can step down to something cheaper.

My rule of thumb: efficiency-first for high-volume, low-stakes work where inference cost and latency matter. Capability-first when a wrong answer is expensive, or when you're still learning what these tools can even do.

That second case matters more than people think. If you start on a weak model while you're new, you'll conclude the technology can't do something when really your prompt was thin.

4. How Much Does It Need to Remember? (Context Window)

The context window is simply how much text the model can hold in view at once. It's counted in tokens rather than words - a token is a chunk of text, roughly a word-fragment, which is why "how many pages fits" never has a clean answer.

Think of it as a desk. A small desk is fine for one letter. A big desk is what you need when you're cross-referencing a contract against three emails.

A practical ladder, no numbers required:

A single email or a short post - almost nothing. Any model handles it.
A long report, a contract, a council planning document - you need a lot of room, and you'll feel it if you don't have it.
A whole year of meeting notes - you need either a very large window or, more likely, a different approach: split the job into chunks, or retrieve only the relevant passages and feed those in.

Two warnings. A bigger window generally costs more, because you're paying for the tokens you push through it. And a bigger window is not automatically more accurate - stuffing a model with 200 pages when the answer lives on page 12 can make results worse, not better.

Notice that Azure's guidance treats context window size as a filter criterion, not a score to maximise. Big enough for the job, then stop caring.

5. Where Does Your Data Go? (Privacy, Security and Region)

Four questions, and you can usually answer them off the provider's own documentation in ten minutes:

Where is the data processed? Data residency matters to some Australian organisations and not at all to others.
Can your inputs be used to train the model?
If they can, is there an off switch - and is it on by default or do you have to find it?
Does your employer, industry body or profession have rules about client data? Health, legal, financial and education work very often do.

Azure's decision tree lists security and compliance plus region availability as hard filters. That phrasing is important. It means a model can be the best performer in every test you run and still be completely ineligible for your use.

Check your own organisation's policy before you paste anything sensitive into anything. This article is general information, not legal advice, and a one-line policy breach costs more than a subscription ever will.

6. Hosted Service or Open-Source Model You Run Yourself?

Azure's guidance lists four deployment options: serverless, managed infrastructure, your own infrastructure, or on-device. That maps neatly onto the question people keep asking, which is "should I use the cloud one or run an open-source LLM at home?"

Honest trade-offs:

Hosted (serverless or managed) - newest capability, no hardware cost, someone else handles updates and security patching. You pay per use or per month, and your data leaves your machine.
Self-hosted or on-device - you control the data, it works offline, there's no per-use fee. You pay in hardware, setup time, and usually a real capability gap against the frontier hosted models.

Choose by constraint, not by ideology. If your genuine, non-negotiable requirement is "this must never leave my laptop", on-device is the answer and the capability gap is just the price. If you want the strongest reasoning available with no fuss, that's hosted, and no amount of open-source enthusiasm changes it.

How to Build Your Own 5-Task Model Test in One Afternoon

This is the part that outlasts every leaderboard. Build your own small evaluation set.

Write down five tasks from your own work. Real ones, from your week. Not clever test questions off the internet.
Make at least one a task you already know the correct answer to. This is your lie detector. Without it you're just judging which model sounds most confident.
Run the identical prompt through each candidate. Same wording, same attachments, no helpful tweaking for your favourite.
Paste the outputs into a spreadsheet. Columns: accuracy, tone, speed, refusals, and effort-to-fix. That last one is the sleeper - a brilliant answer in the wrong voice can take longer to repair than a plain one.
Score each 1 to 5. Your judgement, your tasks, your scale.
Repeat once on a different day. Single runs vary. You will occasionally catch a model having an off turn and mistake it for a verdict.

That last step is the home version of a serious principle. Reproducibility is central to how HELM is designed - results are meant to be re-runnable through the open-source framework rather than taken on trust. Your version of that is simply: run it twice before you believe it.

Two or three candidates is plenty. Five is a research project you won't finish.

How to Read a Benchmark Leaderboard Without Being Fooled

Benchmarks are useful. They're just not a ranking of "good".

Before you take any leaderboard seriously, ask three things. What exactly is being measured? Can you see the prompts and the raw responses? Could someone else reproduce the number?

HELM scores well on all three, which is precisely why it's worth reading. A cropped screenshot with no methodology link scores zero on all three, and that's most of what circulates.

Also watch for the gap between the benchmark and your job. A strong average across academic tasks tells you very little about whether a model can write in your voice about your trade.

Matching Common Jobs to Model Types

No names here - just the shape of what suits what.

Writing, Editing and Summarising

Most general chat models are strong at this, which makes it a good place to be cheap. Judge on voice, not vocabulary - the test is whether it sounds like you after one light edit.

Give it a sample of your own writing before you judge. Tone follows examples far more reliably than it follows adjectives.

Coding and Spreadsheets

Here the capability gap between tiers is obvious and worth paying for. The question "best LLM for coding" gets asked constantly, and the honest answer is: the one that gets your stack right, which you'll only learn by feeding it a bug you already solved.

For spreadsheet formulas, test with your real column layout. Generic examples hide the exact mistakes that'll cost you an afternoon.

Research and Fact-Finding

Treat every model as a research assistant who never admits uncertainty. Prefer tools that cite sources you can click, and then click them.

If a claim has no checkable source, treat it as a hypothesis. That habit alone will save you from the single most common AI embarrassment.

Images, Audio and Video

Multimodal models vary wildly by subject matter. Text inside images, hands, specific equipment, local signage - these are where things fall apart.

Test on the thing you'll be generating day to day. If you need product shots, generate product shots, not fantasy landscapes.

Reasoning Through Something Messy

Reasoning-focused models are slower and dearer, and worth it when the problem has steps - working through a quote with conditions, untangling conflicting instructions, planning a sequence.

Ask it to show its working. Not because the explanation is a guarantee, but because you can spot a wrong assumption far faster than you can spot a wrong answer.

When to Upgrade, Downgrade or Use Two Models Together

You don't have to pick one and marry it. Anthropic's documentation notes that models can be combined, and the everyday version of that is simple.

Use a fast, cheap model to triage, draft or summarise. Hand the hard final step to the stronger one.

Draft the email cheaply, polish the contract clause expensively. Summarise 40 reviews cheaply, write the response to the angry one carefully.

The technical name for choosing automatically between models is routing. You don't need to build it to benefit from the idea - you just need to stop treating "which model" as a single lifetime decision.

Signs You Picked the Wrong Model

You rewrite the output more than you would have written the thing yourself.
It invents details - names, dates, figures - that you have to verify every single time.
It refuses harmless requests, and you're now writing prompts to get around the refusals.
It's fast and pleasant on the easy 80 percent and shallow on the hardest 20 percent, which is the 20 percent you needed help with.
Your bill is scaling faster than your usage, which usually means you're pushing far more tokens than the job requires.

Any one of those is a signal to re-run your five-task test. Not to panic-switch - to re-measure.

Should You Pay for a Subscription or an API?

Subscriptions suit steady personal use. You get a polished interface, predictable monthly cost, and no fiddling with an API key.

Pay-per-token suits bursty or automated use, and it makes cost visible per job - genuinely useful if you're building something and need to know what each run costs.

Estimate your monthly volume before you commit either way. A plan you use twice is dearer per use than pay-as-you-go, and a heavy automated workload on per-token pricing can surprise you in the other direction.

One honest caveat: cost per token changes often. Check the provider's own pricing page rather than trusting any article's figures - including this one.

A Note on AI Stocks and 'Investing in the Winner'

General information only. This is not personal financial advice, and nothing here accounts for your circumstances. Speak to a licensed financial adviser before making any investment decision.

There's an idea floating about that picking AI models and picking AI investments are the same skill - find the winner, back it, done.

The lesson from HELM's multiple leaderboards cuts against that. When evaluation itself splits into safety, capabilities and clinical tracks because no single ranking is meaningful, "no single model wins everything" stops being an opinion and becomes a structural observation.

Which makes "picking the winner" a shakier premise than it sounds, in a field where the leader on one dimension is mid-table on another.

Again: general information only, no recommendation of any kind, and worth a conversation with a licensed adviser rather than an article.

Frequently Asked Questions

What criteria should I compare AI models on?

Start with the three in Anthropic's documentation - capabilities, speed and cost - and set them before you start comparing, so you aren't swayed by whatever demo impressed you last. For anything beyond personal use, add Azure's harder filters: task fit, context window size, security and compliance, region availability and deployment strategy. Write your criteria down first. It's remarkable how often the written list disagrees with your gut pick.

Should I start with the cheapest model or the most capable one?

Both are valid strategies. Efficiency-first means starting cheap and fast, then moving up only if it falls short - good for high-volume, low-stakes work. Capability-first means starting with the strongest model, refining your prompts until the output is right, then testing whether you can step down. Choose capability-first when a mistake is expensive, or when you're new and still learning what's possible.

How big a context window do I need?

Enough for the job, and no more. A single email needs almost nothing. A long report or contract needs a lot of room. A year of meeting notes usually needs a different approach entirely - splitting the work, or retrieving only the relevant passages. Remember a larger window costs more and isn't automatically more accurate. Azure treats window size as a filter, not a score to maximise.

How do I test and score models on my own tasks?

Write down five tasks you genuinely do, including one where you already know the correct answer. Run the identical prompt through two or three candidates. Put the outputs in a spreadsheet with columns for accuracy, tone, speed, refusals and effort-to-fix, and score each 1 to 5. Then repeat on another day, because single runs vary. That repeat is the whole point.

Open-source model or hosted API?

Decide by constraint. Hosted gives you the newest capability, no hardware outlay and someone else handling updates - but your data leaves your machine. Self-hosted or on-device gives you data control, offline use and no per-use fee, at the cost of hardware, setup time and usually a capability gap. If your real requirement is "this must never leave my laptop", on-device is your answer and the gap is the price.

Can compliance rules rule out the best-performing model?

Yes, and this trips people up. Azure's guidance lists security and compliance plus region availability as hard filters rather than scored criteria. So a model can win every test you run and still be ineligible, because of where data is processed or what your profession allows. Check your own organisation's policy first - it saves you running a comparison you were never allowed to act on.

Your One-Page Checklist

Write the job in one sentence, with a verb.
Decide what a wrong answer costs you.
Pick efficiency-first or capability-first - and write down why.
Work out the smallest context window that fits the job.
Check data handling, training opt-out, region and your own org's policy.
Choose hosted or self-hosted by your real constraint, not your preference.
Run your five real tasks through two or three candidates, twice.
Score accuracy, tone, speed, refusals and effort-to-fix.
Consider pairing a cheap model with a strong one instead of choosing one.
Re-test when the job changes, the bill climbs, or something new lands.

One thing worth saying plainly, since we're on the subject of trusting sources: at Daily Tech Wizard every story links to its source, and a person approves every post before it goes out. That's the reason our write-ups of new models are worth reading alongside the vendor documentation rather than instead of it.

If you'd like to keep up with new models without wading through release notes, subscribe to the free daily Daily Tech Wizard newsletter using the sign-up form on this page. Short, sourced, human-checked.

Then do the other half yourself: save that checklist, and run your five-task test before your next subscription renewal. You'll either confirm you're on the right tool or save yourself a year of paying for the wrong one.