According to ESOMAR, Market Research has achieved negligible growth worldwide: 0.4% in 2012 and 0.7% in 2013. The situation is even worse when you look at developed markets like Europe, which has shrunk by over 1% in each of the last two years.
By contrast, according to IDC, the Big Data industry has a compound annual growth rate of 20-50% (services towards the lower end and infrastructure at the higher end). This indicates companies crave more insight, better insight, faster insight and above all competitive advantage. They want to exploit their existing data, capture new data and combine it with external data. If they can do that without conducting surveys then they will.
Taking data from the above sources Market Research will be worth around $40bn in 2014 while Big Data will be $16.1bn. If the respective growth rates continue, by the end of the decade Big Data may overtake Market Research
Why has growth stalled?
As most markets exit recession and return to growth you might expect Market Research spending to organically increase. However, traditional survey spending has decreased as other options like neuroscience, passive measurement and big data have matured.
The outlook and plans from the largest agencies support this view. The latest financial reports from Nielsen, Ipsos & GfK acknowledge the slow organic growth. They are looking instead towards new techniques and data sources to return to growth.
Traditional surveys will continue to be invaluable as an unbiased source of data. In some cases it’s the only way to get the data you need. In other cases survey data becomes more useful when combined with new sources of data. Allan Fromen’s post on GreenBook discusses why the advent of Big Data does not mark the death of Market Research. They can co-exist and often benefit each other.
We need to continue to extend offerings beyond survey based research. Providing clients the best option rather than defaulting to a survey; this is New Market Research (NewMR).
How do we deliver NewMR?
We need to acquire new data. This might be:
- Data we generate without surveys, such as placing TV monitoring or passive web measurement devices.
- Data given to us by clients for analysis. Asking client to check what data they may have that could compliment our investigations
- Open data – from Data.gov or OpenStreetMap
- Social data – from Twitter or Facebook‘s APIs (Application Programming Interface) or from aggregators like DataSift
- Third party data bought for a specific case or to help with multiple clients.
We need the infrastructure and tools to process this data:
- How might we use existing tools for analysis
- If the data is unstructured (text, audio, photo, video) it may need coding or machine learning to use in analysis
- Distributed computing, like Hadoop, for analysis of large data sets
Once we have the data and infrastructure we need to acquire new skills to process and derive insight from new data sources:
- Getting data from the web can require programming, working with APIs and http requests
- Data can be incomplete and full of errors, it often needs careful cleaning
- Combining different datasets can be difficult, particularly aggregated data
- Techniques like clustering, regression and exploring data visually can be useful to understand the data.
- Asking specific questions of the data can be harder. An iterative cycle of data acquisition, cleaning, analysis and review can help. It may then be necessary to go and acquire new data or re-frame your hypothesis.
Most importantly this all needs to be combined into a service that creates value for our clients. Building on our analytical abilities and our positions as trusted advisers but exploiting NewMR techniques more widely.
Why we need to act now
The risk of not embracing change is that MR data, for some clients, could become a commodity data source in their analysis. Analysis would then be carried out in house or by other consultancy firms. Fieldwork and delivering raw data is not as profitable as a full service agency.
In coming posts we’ll look at how data science teams can be formed, sources of data, the platforms that might help and techniques to work with this data.