Top 10 challenges for implementing marketing mix models

Marketing Mix Modeling refers to statistical methods that attribute product performance to various marketing efforts. In the article below I describe the 10 most difficult challenges my team deals with when tackling these models. In subsequent articles I will discuss the different model choices along with their associated pros/cons. Enjoy the article and please comment at the bottom when you are finished.

Below are the top 10 challenges faced by modelers of media mix.

1.      State of Data (SoD)

When a company decides to start measuring the effectiveness of each media channel they have to evaluate their current State of Data. In many cases the marketing department was unaware that they needed to track all sorts of internal and external information. In some cases the state of the data is similar to Swiss cheese and modeling becomes hazardous. My recommendation here is to identify what you need now and begin building data marts to house the new information. In the short run you can work on digital optimization since that data is tracked consistently in Google Analytics. You may have to wait on traditional media data to accumulate before you can fold them into the analysis.

2.      Multicollinearity

When marketing activities coincide in time they create collinearity in the model. This means that the input variables move together or share information. My recommendation here is to either remove highly correlated input variables or use principal component analysis (PCA) to remove the information overlap.

3.      No standards of measurement

It’s hard to compare performance in the industry when there is no consensus on how the effectiveness of marketing efforts should be measured. The effects of a marketing campaign can be immediate and long term. In general estimated increment lift is straight forward but estimating repeat sales or delayed customer acquisition is not.Regardless of your definition of impact the gains must be compared against the costs using a performance metric. Typically marketers use one of following two metrics:

• ROMI = Incremental sales on campaign / dollars spent on campaign.
• Effectiveness: Incremental sales from campaign / Impressions.

4.      Lack of transparency

Most marketing mix modeling today is being done by consulting firms on behalf of their corporate clients. All modeling vendors that I have worked with consistently reject requests for details about the models. Instead they call that information proprietary and release only the ROMIs and optimization plans. Without the ability to review the model themselves clients have no means for validating or confirming the accuracy of the models. In this case I recommend that you withhold a portion of locations or time periods from the data set you are sending to the vendor. When they have completed the modeling process send them only the input data for the missing locations/date periods. They will estimate results and you can compare those to the real results that were withheld. This is called hold-out sampling.

5.      Measuring advertising CONTENT

The content of an advertisement is difficult to quantify. There may be different promotional offers, humor, visuals, celebrity, etc. For TV commercials modelers tend to use Gross Rating Points (GRPs) to measure reach and frequency. But two commercials may have the same GRP count but appeal to consumers very differently. In your model they will be given the same weight and thus degrade performance. I’ve had luck here by adding dummy variables for similar promotions.

6.      Dynamic Effects

When we start a new campaign the effect is not immediate. In fact it may take months to gain full effectiveness of a new series of commercials. This is called the wear-in phase of the campaign. As you can guess when a commercial is becoming stale we call that the wear-out phase. During the wear-in and wear-out phases the incremental lift of each GRP is reduced. My recommendation here is to use Adstock which allows for diminishing returns from impressions today into the future. For practitioners out there, I recommend estimating the half-life by trying different decay rates until you get back the highest R-squared when regressing on your dependent variable. You will have to use a lower half-life when in the wear-in or wear-out phases. I suggest starting/stopping a dummy variable near where you think the phases start/stop. Once you’ve identified when the phases begin and end you can cut the half-life in your Adstock during those periods.

7.      Interaction effect between Ads

Different marketing activities have synergistic effects with one another. Running a TV campaign that is matched by a series of billboard deployments can significantly increase the effectiveness of both. You can test for synergies using classification, decision tree, or CHAID methods. Once you’ve identified there exists a relationship make sure to add an interaction variable in the model.

8.      Non Linear Effects

Customer response from some activities is pretty consistent. Generally, if you spend 10% more in Google Adwords on the same campaign of keywords you will probably get a 10% increase in the number of conversions from that campaign. That’s a linear relationship which is easy to model. On the other hand, traditional media like TV and print have a more complex relationship with incremental customer demand. There is research that shows that a minimum threshold of advertising must be reached in some channels to have any effect. Along the same vein, these channels generate almost no incremental demand after reaching a saturation point. This means traditional media is often better modeled with an S-shaped curve rather than the straight linear regression model.

9.      Instability of coefficients

All of the issues described above can lead to instability of the estimated coefficients. This can be the biggest challenge for your modeling team. If you finalize a model that has unstable coefficients you are setting your team up for disaster in the future. What happens when the data is updated, you run the model, and your variables lose significance or their coefficients change sign? Trying to explain that to management will be a tough conversation… I recommend that you create subsamples prior to starting your process. Then estimate the coefficients using different subsamples of data. If the coefficients are estimated consistently then sign off on them.

10.  Corporate Management

When you first implement a media mix optimization strategy the expectations of management may need some management themselves. Most likely the State of Data will initially only provide insights rather than full blown optimization. I recommend that you start with “Test & Learn” plans to confirm these insights. As your State of Data improves you can start tackling larger more complex optimization tasks.

The road to optimized media spend is a treacherous one. However, if you’re reading the road signs I’ve laid out above you’ll gain some traction. Good luck to all the practitioners out there! Please post any questions below.

You can find my articles here on LinkedIn or the Huffington Post where I regularly write about Data Science, Machine Learning, and Team Building. If you would like to read future posts please click 'Follow' and feel free to also connect via Twitter, Facebook, Instagram, and Google+.