Why marketing's best proof is usually survivorship
Somebody has sent you a case study. The numbers in it are correct. The client is real, the campaign happened, and the growth chart is not doctored.
It still cannot tell you what you want to know.
A case study exists because the engagement worked. That is the condition for its being written at all. So a shelf of case studies is a collection assembled by outcome. Reading it tells you what can happen, not what tends to happen. The first is a fact about the world. The second is the thing you are buying.
Selection happens before you ever see the evidence
The clearest illustration comes from Abraham Wald, a statistician at Columbia's Statistical Research Group during the Second World War. Wald was asked where to add armor to bombers, given the damage on the aircraft that came back. His answer inverted the question. The returning aircraft showed where a plane could be hit and still fly home. So the armor belonged where the survivors showed no damage.
Worth knowing before you repeat it. The version with the dotted diagram and the exasperated generals is largely a later invention. Wald wrote a technical memorandum in 1943 on estimating vulnerability from the damage of survivors. The famous story about survivorship bias is itself a survivor of decades of retelling. Fitting, for a story about survivors.
The mechanism transfers cleanly. Marketing case studies get written by the people who ran the engagement. They cover the engagements that produced something worth writing up. Nobody publishes the account that went nine months and stalled. That is not dishonesty. It is what publishing is.
The number that rarely appears is the denominator
Every case study answers one question loudly and another one silently.
The loud question is what happened. The silent one is: out of how many? A firm that ran 60 similar engagements and published four has told you something. A firm that ran five and published four has told you something else entirely. Both pages look identical.
An older practitioner formulation is worth keeping. Look at how many case studies a firm has in proportion to its client count. You will rarely be handed that ratio. The industry's version of transparency is a portfolio of selected engagements. At the sophisticated end, it is a live dashboard for a chosen account. Neither of those is a roster.
The absence itself is information. Denominators are not withheld because they are boring.
What survivorship does to a number
Finance has measured this. Fund performance is one of the few places where the disappeared cases were recorded before they vanished.
Researchers looked at the American domestic equity fund market from 1993 to 2006. They ran it twice, once with the funds that closed or merged and once without. The correction changed the sign of the result. Counting only the funds that lasted, the category looked slightly ahead. Counting the ones that died as well, it was behind by about 1% a year.
Hold onto what that shows. Survivor-only data did not merely flatter the category. It reversed the verdict, turning a market that lost to its benchmark into one that beat it. The bias was not a rounding error on the answer. It was the answer.
Marketing rosters churn faster than fund families do, and no one publishes the ones that left. Whatever the size of the effect in your vendor's portfolio, it runs in the same direction.
Four questions that make a case study useful
None of this means case studies are worthless. It means they need reading. Four questions turn one into evidence you can use.
- What was true before the engagement started? The starting list, the existing audience, the brand's age, the category's search volume, the budget. A result resting on 20,000 existing subscribers is a fact about those subscribers as much as about the work.
- Out of how many? Ask directly. The answer, including a refusal to answer, tells you what the page cannot.
- What else changed in the same window? Funding, a new product, a hire, a competitor's collapse, a seasonal swing, a category-wide shift. Marketing rarely runs as the only variable. A firm that names the other variables is showing you it can think.
- Can I check it? A named client, a contactable reference, and a public property you can look at yourself. The Internet Archive holds versions of most sites going back years. You can see what the client's pages looked like on the dates the study describes. Coverage has gaps, since some publishers began blocking the crawler. It will still settle a timeline more often than not.
The fourth question is worth more than it used to be. Producing a convincing case study now costs an hour and no clients at all. That makes the verifiable ones more valuable.
Run it on mine
I publish a case study. It describes a business coach whose traffic grew by orders of magnitude over years of work. That client reached first place nationally for the terms that mattered in the field.
Here is what it cannot tell you.
It is one engagement. My denominator is small, because I take few clients at a time. One engagement is not a distribution, and I will not present it as one.
That client arrived with deep expertise and a working business. Those starting conditions were not mine to create, and I cannot promise you have them.
Several things changed over those years that were not my doing. The field grew and search behavior in the category shifted. The client also kept showing up for interviews, year after year.
The part I put on the page deliberately is the part that stayed hard. Traffic grew and converting that traffic into booked appointments remained difficult throughout. A case study that reports only the graph going up is hiding the most useful thing in it.
That is the standard I would want applied to me, so I apply it to myself in public. It is also the reason the first question about any marketing claim is who benefits from your believing it.
When there is no base rate to be had
Sometimes the answer is that nobody knows the rate. Marketing engagements sit in no public database, and no regulator collects the outcomes. The firms holding the data have no reason to publish it.
That leaves four substitutes, in the order I would use them.
Check whether the mechanism makes sense for your business rather than whether it worked for someone else's. Check whether the conditions behind the winning case are conditions you have. Buy in pieces small enough that being wrong is survivable. And talk to a client who is two years in. The ones still there after two years are a better sample than the ones featured at launch.
That is weaker evidence than a base rate. It is what exists.
The limit of this kind of skepticism
Taken too far, survivorship reasoning becomes a way of never deciding anything. Every proof gets dismissed and every vendor gets treated as a liar. The owner sits still while less careful competitors move.
So hold the point precisely. A case study is evidence of two things. The outcome is achievable, and the firm has done work of that kind at least once. Both matter. Neither is a probability.
The next case study you are handed is probably true. The question worth asking is what it is true of.
