Almost every discussion of this topic begins with the wrong vocabulary, so it is worth fixing that first. Very little of what gets called open source AI actually is.
The vocabulary problem
Traditional open source means you get the source, you can modify it, and you can redistribute it. Applied to a model, the equivalent components would be the training data, the training code, and the resulting weights. Almost nobody releases all three.
What is usually released is open weights: the trained parameters, downloadable and runnable, often under a licence with restrictions. That is genuinely valuable — you can run it on your own hardware, inspect its behaviour, fine-tune it, and nobody can take it away. It is not open source in the conventional sense, because you cannot reproduce it from scratch and you frequently cannot use it for anything you like.
The three tiers actually available:
| Tier | What you get | What you cannot do |
|---|---|---|
| API-only | An endpoint, documentation, a bill | Run it yourself, inspect it, keep it if it is withdrawn |
| Open weights | Downloadable parameters, freedom to run and fine-tune | Reproduce it; sometimes compete with the publisher, depending on licence |
| Fully open | Weights, training code, data description, permissive licence | Little, but these are rarer and usually smaller |
Most real decisions are between the first two tiers.
Where each is genuinely better
Closed API models
Peak capability. The strongest models at any given moment are generally API-only. The gap has narrowed considerably and it moves constantly, but at the frontier it is real.
Zero operational burden. No GPUs to buy, no serving stack, no capacity planning, no 3am page because the inference cluster fell over. For a small team this is decisive and frequently underweighted.
Elastic cost. Ten requests one day and ten million the next costs proportionally. No idle hardware.
Someone else's problem list. Safety tuning, abuse handling, uptime, optimisation. All included.
Open-weight models
The data never leaves. The strongest argument, and often the only one that matters. If you handle medical records, legal files or anything under strict residency rules, running locally may be the only lawful option. No contractual assurance is equivalent to the data not moving.
Predictable cost at scale. APIs are cheap per call and can be very expensive at volume. If you run steady high throughput, owned or rented hardware is usually cheaper — often dramatically so, because you are no longer paying a per-token margin.
Real customisation. Full fine-tuning on your own data, not just a hosted adapter. For narrow domains with unusual vocabulary this can make a small model outperform a much larger general one on your specific task.
It cannot be deprecated. An API you build on can change behaviour, raise prices, or be retired. A model on your storage behaves the same in five years. For anything requiring reproducibility — regulated decisions, research, audit trails — this matters enormously.
Latency floor. Local inference has no network round trip. For interactive applications that is worth tens of milliseconds.
Choosing open weights to save money, without costing the engineering. A serving stack needs batching, quantisation, autoscaling, monitoring, failover and someone who understands all of it. That is one to three engineers. Unless your inference bill exceeds their salaries, the API is cheaper — and the crossover point is higher than most teams estimate.
The four questions that decide it
1. Can the data leave your infrastructure? If regulation, contract or policy says no, the decision is made. Everything else is secondary.
2. Is your load steady and high, or spiky and low? Steady and high favours self-hosting. Spiky and low favours the API, because idle hardware is pure waste.
3. Do you need the frontier, or is good enough good enough? For summarisation, classification, extraction and routine drafting, a mid-sized open model is entirely adequate. For hard reasoning, long-context analysis or difficult code, the frontier still earns its price.
4. Who operates it? If the answer is "nobody has time", choose the API regardless of what the cost spreadsheet says.
The pattern most mature teams land on
Not one or the other. A router.
- Small open model, self-hosted, for the high-volume, low-difficulty work: classification, tagging, extraction, boilerplate. This is usually 80 to 95 per cent of request volume and it is where the money is.
- Frontier API model for the small fraction of requests that are genuinely hard, or where quality is visible to a customer.
- A local model as fallback if the API is unavailable, so the product degrades rather than fails.
- Strict data classification determining which path a request may take, enforced in code rather than by convention.
This is more work than picking one, and it is where the actual cost savings live, because it stops you paying frontier prices for trivial requests.
What self-hosting genuinely requires
If you go this way, budget honestly.
- Hardware. Enough accelerator memory to hold the model plus the key-value cache for your concurrency. This, not raw compute, is usually the binding constraint.
- A serving framework with continuous batching. Naive serving wastes most of your hardware.
- Quantisation. Reducing numerical precision cuts memory substantially with modest quality loss. Almost always worth doing, and it needs evaluating rather than assuming.
- Evaluation. Your own test set, run on every model or configuration change. Without this you cannot tell whether an upgrade helped.
- Monitoring. Latency percentiles, throughput, memory headroom, output quality sampling.
- A person. The recurring cost people forget.
Licence traps worth reading for
Open weights does not mean unrestricted. Common clauses:
- Acceptable use policies prohibiting specific applications.
- Scale thresholds that change the terms above a user count or revenue figure.
- Competition restrictions barring use in products that compete with the publisher.
- Attribution and naming requirements for derivatives.
- Restrictions on training other models on the outputs.
If your business depends on the model, have someone read the licence properly. "Open" on a download page is a marketing word, not a legal one.
The honest state of things
Open-weight models have improved faster than most people expected, and for the majority of production tasks the quality question is settled: they are good enough. The frontier remains ahead on the hardest problems, and probably will for as long as training the largest models requires the largest budgets.
What that means practically is that the choice has stopped being about capability and become about operations, economics and data governance. Which is a much more tractable conversation, and one your organisation is probably already equipped to have — it is the same conversation as buying software versus running it, which has had the same four answers for thirty years.