Why Most Agency Incrementality Tests in 2026 Are Decorative
Most geo holdouts run by agencies on mid-market accounts cannot produce a statistically significant result inside a 4-week window. The client signs off because nobody in the room wants to admit the test was theater. The math is unforgiving. The slides are not.
Incrementality testing has gone mainstream — adoption is up across both brand and agency-run programs, and most performance shops now include some form of lift study in their measurement pitch. Rigor has not kept pace. That gap is what this piece is about.
Quick glossary, because the words get used loosely:
- A geo holdout turns ads off in some markets, keeps them on in others, and compares the difference.
- A conversion lift study is the platform-native version, where Meta or Google randomly withholds ads from some users and measures the gap.
- A media mix model (MMM) is the statistical alternative when you can’t run an experiment at all.
The rigor problem is not which one you pick. The problem is that agencies pick the geo holdout by default, on accounts where it cannot work. Platform-native lift studies have matured enough on both Meta and Google to be the honest fallback for mid-market accounts — that changes the answer for most of your roster.
Five Tells That Your Agency’s Incrementality Test Is Decorative
A decorative incrementality test is one designed to produce a slide, not a decision. It runs, it concludes, the agency presents “directional findings,” and nobody acts on the result. Here are the five tells an operator can use to audit a test plan before it runs.
1. No pre-period parity check. Before the test starts, the agency should show you that the test geos and the control geos tracked closely on the outcome metric (revenue, leads, calls) for the preceding 8 to 12 weeks. If that check is not in the test plan, the post-test comparison is meaningless. You cannot measure a lift if the two groups were not comparable to begin with.
2. No minimum detectable effect disclosed before launch. The minimum detectable effect (MDE) is the smallest true lift the test can actually prove at 95% confidence. The agency can calculate it before the test starts, based on your spend, your conversion volume, and the planned duration. If they will not give you an MDE number up front, the test can detect any lift it wants to detect, after the fact.
3. Holdout under 15% of geos. Tiny holdouts are easier to sell to clients because they sacrifice less revenue. They are also statistically useless. A 5% holdout on a national account produces a control group too small to compare against anything. If the agency proposes a 10% holdout, ask what MDE that produces and watch the response.
4. No contamination plan for adjacent markets. If you turn ads off in Sacramento but they keep running in San Francisco, viewers in Sacramento still see brand exposure spilling over from Bay Area TV, search, and social. A real test plan names the adjacent-market contamination risk and says how it is being controlled.
5. No pre-committed action tied to specific iROAS thresholds. Before the test runs, the agency and the client should agree: at an incremental ROAS (iROAS, the actual lift divided by ad spend) above X, we scale. Below Y, we reallocate. Between X and Y, we hold and re-test. If the action plan is not written down before launch, the post-test conversation becomes about defending the prior decision, not making the next one.
Operator Note: “We’ll learn something directional” is the most common phrase in a decorative test. Real tests learn something binary: act, don’t act, or re-test. Directional is the language of a slide.
incrementality testing for agencies options compared side by side.
At What Monthly Spend Does a Geo Holdout Actually Produce a Real Read?
In our experience, below roughly $40,000 per channel per month, a 4-week geo holdout rarely reaches statistical significance on a realistic lift. The honest move is a platform-native conversion lift study or nothing. This is the threshold the rest of the agency conversation should be built around. It is an operator’s heuristic, not a published constant. Run the MDE math on your specific account and you may move it 10K in either direction. The order of magnitude is the point.
Here is the plain-English version of the math. The smaller the account, the fewer conversions, the larger the noise floor. The smaller the account, the larger the true lift has to be before you can prove it exists. Realistic incremental lifts for most paid channels sit well below the threshold a small, short test can detect. Below the spend floor, your test can only detect very large lifts. Anything smaller disappears in noise. The test comes back inconclusive and gets re-framed as “directional.”
The critical distinction most agency test plans get wrong: the floor is per channel, per month, not total account spend. A client running $120K/month across Google Search, Meta, and YouTube is not running a $120K test. They are running a $40K test on each channel, three times. The MDE math treats each channel separately because that is what you are measuring.
Key Concept: Per channel, per month means the spend going through one auction system, on one platform, in one calendar month. A geo holdout on Meta tests Meta. It does not test Google. If the agency is reporting a single account-level iROAS from a single geo test, the test is measuring something, but not what they’re claiming.
The Method Matrix: Geo Holdout, Conversion Lift, or Nothing
The right method depends on the per-channel spend tier. Here is the decision matrix we use before recommending any test design.
Per-channel monthly spendRecommended methodTest windowWhat it actually proves
Under $40K
Platform-native conversion lift OR no test
2 to 4 weeks
Whether the channel produced ANY lift (binary read)
$40K to $80K
Geo holdout with 8+ week window
8 to 12 weeks
A directional iROAS range, not a precise number
Above $80K
Geo holdout, 4-week window viable
4 to 6 weeks
A point-estimate iROAS with usable confidence
Meta Conversion Lift runs a ghost-bid experiment. Meta keeps a randomized control group out of your ad delivery and compares conversion rates to the exposed group. You pay for the held-out impressions in foregone revenue, not in a vendor invoice. Per Meta’s Conversion Lift documentation, the study requires sufficient audience size and conversion volume. Small accounts may not qualify.
Google Ads Conversion Lift works the same way for Search, YouTube, and Display. Per Google’s incrementality documentation, the platform offers both user-level and geography-based conversion lift, with the geo version available for larger campaigns using any data source. Eligibility still depends on volume — check the in-platform requirements before promising a client a study will run.
A homegrown geo holdout is the right answer only when both platform-native options can’t run (cross-channel test, offline conversions, channels without native lift studies) AND the spend tier supports the MDE math. Vendor platforms like Measured automate the geo-test design for enterprise accounts, but the spend floor still applies. The tool does not change the statistics.
An MMM is the right answer when the client wants “what is each channel really worth” across a full media mix and cannot take any channel offline for a test. It is not an experiment. It is a regression model on historical data. Call it that when you sell it. Our Meridian MMM dashboard build walks through how that looks in practice.
How Do I Present a Sub-1.0 iROAS to a Client Trained on Last-Click ROAS?
Lead with the dollar reallocation, not the lift ratio. A CMO who has measured every campaign on last-click ROAS for ten years will read an iROAS of 0.8 as “the ads don’t work.” That is not what 0.8 means, and it is not what you are recommending. You have to give them the language to defend the next decision internally.
The script we use in client readouts looks roughly like this:
Last-click attribution credited this campaign with $400K in revenue last quarter. The test tells us $180K of that would have happened anyway, through brand search, direct, and organic. The incremental piece is $220K, which is real money but lower than the platform claimed. Here’s the $220K we’d move and where it goes.
Notice what is missing: the words “iROAS,” “0.8x,” and “underperforming.” Notice what is present: a dollar amount, a destination, and a recommendation. The client can take that to their CFO. They cannot take a lift ratio to their CFO.
The political reality is that the client hired you on a last-click ROAS number, and now they have to defend a different one to whoever signs the renewal. Your job is to make that defense easy. The reallocation framing does that. The iROAS-as-grade framing kills the renewal.
The other piece of this conversation is the contradiction problem. The platform UI still shows the old ROAS number. Smart Bidding still optimizes toward the old conversion signal. When the client logs into Google Ads on Monday, they will see numbers that disagree with your test. Address it before they ask. Frame the platform number as the attributed read, the test as the causal read, and explain that the two are measuring different things on purpose. Our piece on last-click attribution in the AI Mode era covers the longer version of that conversation.
How Do I Run Incrementality Across a Roster of Clients?
The sequencing rule for a multi-client agency: test the channel with the largest reallocation risk first. Reallocation risk means the channel where being wrong costs the client the most monthly budget. That is almost never the channel that is easiest to test.
The agency-ops problem most testing guides ignore is that you don’t run incrementality on one account at a time. You run it across 10 or 20 or 40 accounts, each with different budgets, different sophistication levels, and different patience for held-out revenue. Here is how we sequence.
- Tier 1, above the spend floor. Bake incrementality into the retainer as a semi-annual readout. Sequence by reallocation risk. The first channel tested is the one with the largest budget where the platform-reported ROAS feels suspiciously high.
- Tier 2, at or near the spend floor. Project-priced annual lift study using the platform-native option. Don’t promise a precise iROAS. Promise a directional read on whether the channel is producing any lift at all.
- Tier 3, below the spend floor. Don’t run a test. Tell the client what they actually want is an MMM, an A/B creative test, or a structural account review. The honest no preserves the relationship better than a decorative yes.
The billing question for inconclusive tests deserves a clear answer in the SOW before anyone signs. Two options work cleanly:
- A re-run at agency cost if the inconclusive read was caused by an avoidable design flaw (bad geo matching, holdout too small, contamination).
- A credit toward an MMM engagement if the inconclusive read was caused by insufficient spend for the method.
“Directional readout” is not a remedy. It is the language of a decorative test, and it trains clients to accept theater.
On cadence, semi-annual is enough for stable accounts. Quarterly fits accounts with rapid budget shifts, seasonality, or major creative refreshes. Never on demand from a nervous client. A test run reactively because something looks wrong this week is a test designed to confirm a panic, not to produce a real read.
Frequently Asked Questions
At what monthly spend does a geo holdout actually produce a real read?
In our experience, roughly $40,000 per channel per month is the floor where a 4-week geo holdout has a realistic chance of detecting a modest lift at 95% confidence, and even that requires good geo matching and a clean pre-period. Below that, the MDE math forces you to detect lifts so large most channels don’t actually produce them. The honest answer for smaller accounts is a platform-native conversion lift study or an MMM. The threshold is per channel, not total account spend.
When should I use Meta Conversion Lift vs. Google Ads Conversion Lift vs. a geo holdout?
Use the platform-native lift study when you’re testing a single channel and your spend on that channel is below the geo-holdout floor. Per Google’s incrementality documentation, both Meta and Google offer ghost-bid based conversion lift studies that handle single-channel measurement natively. Use a geo holdout when you need to test cross-channel effects, when offline conversions are part of the outcome, or when the platform-native option doesn’t cover the campaign type. Use neither when the client’s real question is “what should my media mix look like overall.” That’s an MMM.
How do I present a sub-1.0 iROAS to a client trained on last-click ROAS?
Lead with the dollar amount you’re recommending to reallocate, not the lift ratio itself. A 0.8 iROAS reads as “ads don’t work” to a last-click-trained CMO unless you frame it as a budget move: platform attribution said X, the test says Y is incremental, here’s the gap and where we’d put it instead. Give the client the language to defend the reallocation internally to their CFO. The iROAS number itself never goes to the CFO.
How do I bill for an incrementality test that comes back inconclusive?
Pre-commit the answer in the SOW before the test runs. Two options work cleanly: a re-run at agency cost if the inconclusive result was caused by an avoidable design flaw (bad geo matching, holdout too small, contamination), or a credit toward an MMM engagement if the inconclusive result was caused by insufficient spend for the method. “Directional readout” is not a remedy and trains clients to accept theater. Writing the answer into the SOW protects the relationship when the test doesn’t produce a clean read.
How often should an account be re-tested for incrementality?
Semi-annual for stable accounts above the spend floor, quarterly for accounts with rapid budget shifts or major creative refreshes, and never on demand from a nervous client. A test run reactively because the platform numbers look wrong this week is a test designed to confirm a panic, not to produce a real causal read. Build cadence into the retainer so it happens on schedule, not in response to noise.
Audit Your Next Holdout Before You Run It
If your agency is about to run an incrementality test, or just finished one, bring it to us before you act on the read. We’ll walk through the spend floor, the MDE check, and the decorative-test checklist with your team and tell you whether the test will produce a real answer before you spend the holdout budget. It’s a working session, not a sales pitch. Book a free consultation with Elevarus and we’ll look at the plan.