For all the benefits and promise, marketing mix models can crumble under the weight of the decision. When this happens, it causes the decision maker to resort back to using their own and less-informed mental model. It also leaves everyone involved in the process to wonder why they expended such effort, time and money to go through the process. We commonly see three reasons why marketing models fail: mismatched frame, improper use, and missed timelines.
Mismatched Frame occurs when the decision maker desires to make a decision about a tactic that is not in the model. This commonly occurs when there are new tactics under consideration. For example, imagine a manager who wants to invest in television for the first time after having previously invested in digital tactics such as paid search. It’s common for marketing mix analysts to restrict their model to only the historical data, rather than the frame for the future decision. In other words, the model is framed in service of the historical data rather than in service of the decision maker.
A closely related cause for mismatch is that the statistical analyst, desiring to follow statistical practices, will remove variables from the model that are not “statistically significant” according to the data. If there has not been any discussion with the decision maker about the frame, then removing this variable will limit the possible frame. In this case, the analyst relying on the data, is creating the decision frame rather than relying on the decision maker for the frame.
Further, it’s possible for estimates from the model to defy the logical expectations of the tactics. For example, it’s common for some marketing estimates to be negative when there are problems with multiple correlation or for other reasons for price to be estimated as positive. If left in the model, these results can oppose common knowledge and more importantly, oppose the prevailing mental model of the decision maker. If removed from the model, then this can lead to the frame mismatch problem.
Improper Use occurs when the model is used in a way that does not match the natural conclusions of the model. The most common cause of this is reliance on ROIs and Contributions as metrics to inform future decisions. It’s important to understand that Contributions from marketing mix models provide the decision maker with an estimate of the difference between investing at a given level and investing nothing. ROIs typically reflect a translation of Contributions and Investments into financial terms. Literally, ROIs provide an estimate of the value of investing at the simulated level versus investing nothing. This implies that the only two choices are investing at the simulated level and investment nothing, which does not express the idea that investments can be incrementally increased or decreased.
Expanding further, if the decision maker were to consider investments at lower or higher levels, they would be left without information from ROIs or Contributions alone. Unfortunately, many decisions are erroneously made assuming that the stated ROI holds up for any level of investment. If the ROI is low the tactic might be cut all together or, if high, seen as a blank-check endorsement for more investment at any level. Both are erroneous conclusions because they ignore the fact that marketing investments have continuous and diminishing marginal returns. ROI-based decision rules ignore the knowledge that returns increase by spending less and decrease by spending more, and that there exists a financially optimal investment for each tactic.
Missed timelines happen with marketing mix modeling projects when those involved in the effort fail to appreciate all the problems with Complexity, Conflict, Cost, and all the ways in which marketing mix models can fail. This leads to endless iteration and rework. Many marketing mix modeling projects are planned to be eight-week efforts only to turn out to be sixteen-week slogs.
One of the hidden costs to timelines, especially with those who focus on the statistical modeling aspects, is ambiguity. As a part of the upfront marketing data analysis and collection effort, there is typically a lot of ambiguity in the way that certain tactics are defined and monitored. For example, is Influencer a tactic unto itself or is it a social media tactic? What does it mean to invest in Influencer tactics? Just wrestling these questions to the ground can take a week by the time all the people are consulted. Further, these questions may not even be raised until after the first iteration of the model is completed.