All right, so you’ve made that large pricing strategy shift. You spent days crafting a price plan, with forecasts, projections, margin estimates, etc. But, once the change is approved and pushed live, the team is now moving on to the next pressing issue. The decisions they made were good... right?
Unless something materially breaks, no one is asking "Did that work? Was our promotion too steep? Did one category perform better than the other, and why?"
However, measuring a pricing strategy’s impact to your business is as important, if not more than making the right decisions upfront. Measurement allows you to catch bad decisions, observe them in production, and correct them before they meaningfully damage your business.
This blog post attempts to answer the question of how do you measure and attribute business impact of a pricing strategy. If you haven’t already read it, do read our previous post how can you do price optimization in retail, without running user A/B tests first.
The best way to measure the impact of a pricing strategy without an A/B test is to rely on the presence of Control SKUs, or products where the price change was not applied. The performance of the SKUs where a pricing change was made, or the Tested SKUs, can be compared with these Control SKUs by using an old workhorse in econometrics known as the differences-in-differences model (which you may see as "diff-in-diff" or “DID”).
The goal of the DID is to ask the counterfactual question - “How would the Tested SKUs have performed if we hadn't rolled out our pricing strategy” and compare that against how the Tested SKUs actually performed under our pricing plan.
This counterfactual has been represented by the dotted line in red below. This counterfactual is compared against the “actual” performance of the Tested SKU, represented by the blue line. The delta between the blue line and red dotted line produces the business impact.
The Control SKUs, represented by the green line, are used to calculate the counterfactuals (red line) by assuming a constant difference between the groups, before and after the change.
There’s some key conditions that need to be met for this model to be valid. One you can immediately see from the graph – the two trends between control and treatment SKUs have to be parallel, i.e. the difference between the two must be fixed in the absence of our price recommendation. If there are time-varying differences between the two, this approach isn’t valid.
Another subtle way in which this could occur is if there’s spillover in demand between the two groups. That is, suppose the tested SKUs both belong to the same product category, like ice cream. If we decrease prices on treatment ice cream, treatment ice cream would cannibalize demand from the control ice cream. We would mis-attribute the impact of our price recommendations on demand as being higher than they should be – and inevitably be disappointed when we roll out our price recommendations on all products and find the boost in demand is smaller than expected. This problem is actually relatively simple to address – make sure that treatment and control SKUs are typically unrelated products, e.g., assign ice creams to treatment and soda to control.
(In case you’re wondering, this spillover issue is an example of a violation in the “stable unit treatment value assumption” – a mouthful abbreviated as “SUTVA”)
As you can tell, there is a ton of complexity baked into the simple diagram we've shared, and in practice we carefully customize our approach to each client to capture these potential complexities. If you want any input or further direction, book a call with us to chat about how we can help.
There’s a fairly active literature on this general topic – here’s a small selection of additional resources:
