If you accept https://medium.com/@ranjeettate/how-should-one-evaluate-incremental-return-on-ad-spend-d578be43ecd8, then we agree that it is critically important to get the incremental revenue (due to the ad campaign) correct. We know what the revenue is after the campaign, so what we need is what the baseline revenue would have been had the ad campaign not run. This is done by some combination of forecasting, modeling and testing on a control group. AI based attribution modeling and analysis even gives you all the cross-media (TV, TV + display etc.) lifts in various metrics (e.g. brand awareness, familiarity, favorability, consideration …) . See https://www.iab.com/wp-content/uploads/2017/01/IAB-Cross-Media-Ad-Effectiveness-Study-Jan-11-2017.pdf for example. However, such studies seem to miss out on a couple of things:
First, while they acknowledge the differing costs of advertising on different media (TV is much more expensive than mobile which is more expensive than display), their analysis doesn’t seem to take cost into account. I would think that this is easy enough, by calculating lift per dollar spent for each metric. Similarly, the study is great in recognizing discrepancies between actual and “optimal” frequencies of digital media advertising, but is optimizing lift instead of lift/$. If lift per dollar were considered instead, the optimum frequencies and budgets would be much lower than they would be for optimizing lift. Also, since TV advertising is more expensive, the relative “lift per dollar” effectiveness of cheaper digital media would be greater, and the optimum media mix would be different.
The next two points are particularly important for digital media advertising. A digital ad serves, in my mind, three functions: i) branding, through exposure to the creative, ii) targeting, since each ad is only shown to one person at a time, which is where the Machine Learning comes in, and iii) a direct entry into the purchase stream.
Here are a couple of different ways I think about this, via “ablation studies”, and please feel free to disagree or add to it:
Advertising on traditional broadcast media only serves the first function: that of increasing branding (I always imagine this as a logo on a red-hot poker searing my brain.). Personalized TV like Roku etc. additionally serves the targeting function, but doesn’t yet provide an entry point into the purchase funnel.
If the display ad had no link or if the link were broken, it would serve the branding and targeting function, and at best be like a hoarding or TV ad exhorting you to “Call 1–800-BAR-ROWS to get your discounted wheel barrow today!”. For the following, I’m going to assume that the link works.
Finally consider an “ad” which consists of only a link to a eCommerce website, with no branding or targeting. This function provides people who were already in consideration easy access to make the purchase. It certainly increases the probability of purchase and hence overall revenue, but clearly this is independent of any branding or targeting.
Second, while the analysis parses out the contributions of different media and mixes, it completely ignores the existence (effectiveness and cost) of the underlying creative. If I made a beautiful ad and all the credit for the ROAS went to the AdTech company, I would be mad. While you can “easily” A/B test the relative effectiveness of different creatives on the same medium, without a means of separating the effectiveness of the creative from the effectiveness of the medium it will be impossible to do a true cross medium comparison.
Third, the fore-mentioned analysis doesn’t measure targeting effectiveness. There do exist analyses that calculate incremental ROAS and there are models for optimizing overall incrementality, but since they don’t seem to separate out the effectiveness of the creative from the targeting they can’t be used to calculate the part of the ROAS contributed by the targeting technology.
But because there is lots of data in digital advertising, there is a “physics” way of measuring
- the baseline revenue (the counterfactual corresponding to what the revenue would have been without the advertising campaign, without resorting to PSAs or holdout groups and ensuring those are bias free).
- the contribution of the creative to the ROAS, and
- the contribution of the targeting technology to the ROAS.