Method
How to read an agency case study
Claims currently live on agency roundups in this market include 30x ROAS, 95% client retention and 1,980% user growth. Not one of them states a sample, a period or an attribution basis. Here is how to work out what is left when you take those away.
- Checks
- Five
- Applied to our own claims
- Yes, at the end
- Case studies on this site
- None
Every number needs a denominator
“We generated $10 billion in revenue for our clients” is not a claim about performance. It is a claim about how long the agency has existed and how many accounts it has touched. “30x ROAS” is not a claim about skill until you know whether the account was scaling or being harvested. “95% client retention” means nothing without knowing over what period, and whether it counts clients who reduced their spend to nothing but never formally left.
None of this makes the agencies liars. It makes the numbers incomparable, which is convenient in a market where every competitor publishes something similar. The five questions below are the cheapest way to find out which kind of number you are looking at, and all of them can be asked in a first call.
01
The five checks
In order. The first two disqualify most claims on their own.
1. Ask for the period and the spend level, not the multiple
A 12x return on $4,000 a month and a 3x return on $400,000 a month are not the same achievement, and the second is far harder. Efficiency almost always falls as spend rises, because you exhaust the cheap demand first. A case study that gives you a multiple without the budget behind it has removed the only context that makes it mean anything.
2. Ask what the number was before, and for how long
“We achieved a 4x ROAS” is a statement about a level. The question is the change: what was it in the ninety days before, measured the same way? An account that was already at 3.8x did not need an agency. An account that was at 1.2x and is now at 4x has a real story, and the agency should be glad to tell it.
Watch for the baseline being a single bad month. Compare quarters, not months, or you are mostly measuring seasonality.
3. Ask which attribution setting produced it
The same account will report materially different returns under last-click, data-driven, a seven-day-click window, a one-day-view window and SKAdNetwork-modelled results. All of those are “the platform’s number”. Changing the setting changes the result without anything happening in the business.
Ask the follow-up too: does that number include branded search? Branded traffic converts at several times the rate of everything else and would have converted anyway. Any agency that reports blended numbers with brand folded in is reporting your existing demand back to you as its own work. Splitting branded from non-branded is the single most clarifying thing you can do to a paid search report.
4. Ask what happened after the case study ended
Case studies are written at the peak. The interesting question is what the account looked like six months later, and whether the client is still there. If the answer is that the relationship ended shortly after the screenshot was taken, that is worth knowing, and the reason may be entirely innocent.
5. Ask for one account that did not work
This is the most revealing question in the list, and the one people are most reluctant to ask. Everyone has failed accounts. An agency that cannot produce one either has very few clients or is managing the conversation. What you are listening for is whether they can describe what they got wrong without relocating the blame onto the client, the market or the platform.
02
Why the platform number and the business number diverge
Not fraud. Two systems built to answer different questions.
An ad platform counts a conversion when it saw a click, or sometimes a view, within its own window. Each platform does this independently, so on a purchase preceded by a Google click, a Meta impression and an email, all three can claim it. Add them up and you get more revenue than the business made. Your own database counts it once.
That does not make platform figures useless. They are the feedback loop the bidding system needs, and without them automated bidding cannot function. They stop being useful the moment they are presented as a business result, because they have no way of knowing what would have happened if the ad had never run.
03
The same five checks, applied to us
This site has no case studies to test, so here are the claims it does make, with what each one does and does not prove.
| What we claim | What it proves | What it does not prove |
|---|---|---|
| 200+ ad accounts since 2019 | Volume of exposure across categories and budget sizes | That any of them performed well. It is a count, not a result |
| Budgets to $1M+ a month | That the largest account we have run was at that scale | That the typical account is. Most are far smaller |
| 14 clients past one year, 5 past two | Clients kept paying after the initial period, which is a decision they made with their own money | Their results. Tenure and performance correlate, but inertia exists too |
| Longest relationship 63 months | One relationship lasted that long | Anything about the median, which is the number you would actually want |
| 10 years in paid acquisition | Time in the discipline | That the last three years resemble the first three. The platforms changed underneath |
Source: Every figure here comes from our own books, and the same figures appear everywhere else on this site because they are stored in one place. None has been audited by anybody, and we would treat that caveat as disqualifying if a competitor offered it.
Put those through the five checks and the honest summary is this: we publish tenure because tenure is the one thing we can evidence without naming a client, and tenure is a weaker signal than results. It is a decision clients made with their own money, repeatedly, which is not nothing. It is also not a performance claim, and we would rather say so than let it read like one.
There is no median in that table because we hold too few accounts in some categories for a median not to identify a client. That is a real limitation of publishing anything under an NDA, and it is explained at greater length on why there are no case studies here.
The thing we would rather be judged on is the audit, because it is the only artefact where you can check the reasoning rather than the result.
Questions people actually ask
- What is a good ROAS?
- There is no answer without your contribution margin. A 3x return is comfortable on a 70% margin subscription and loss-making on a 25% margin retailer once shipping and returns are counted. Anyone who answers this question with a number, without asking about your margin first, is not measuring your business.
- Why does the agency report a different number from my own system?
- Because they measure different things. Platforms count a conversion if they saw a click or view within their attribution window, and each platform does it separately, so three of them will happily claim the same sale. Your own database counts it once. Treat platform figures as steering data and your database as the scoreboard.
- Is platform-reported ROAS useless then?
- No, it is useful for the job it does: telling the bidding system which of its own decisions worked. It stops being useful the moment it is presented as a business result, because it has no idea what would have happened if the ad had not run.
- How do I test whether the ads caused the sales?
- A geo holdout is the cheapest honest test. Turn the channel off in a set of comparable regions, leave it on elsewhere, run it for at least four weeks, and compare the change in total sales rather than in platform-reported conversions. It is blunt, it costs real revenue while it runs, and it is still better than arguing about attribution models.
- What does a trustworthy case study look like?
- It states the period, the spend level, what the number was before, which attribution basis produced it, and ideally what happened in the six months afterwards. Very few do. The absence is not always dishonesty, but it does mean the number cannot be compared with anything.
- Should I just ignore case studies?
- Read them for the reasoning rather than the result. A case study that explains what was diagnosed and in what order tells you how the agency thinks, which is the thing you are actually buying. The multiple at the end tells you very little.
Judge the reasoning, not the multiple
The audit is a document about your own account: what we would change, in what order, and what we would leave alone. It is free, it takes about five working days, and you can act on it without us.