bill vorhiesSummary: Over 80% of companies are not yet using advanced analytics. Here’s a step-by-step plan to implement a brand new predictive analytics program getting the biggest bang for your buck from the most cost effective investment. Part 2.

In Part 1 (link here) we covered our fundamental strategy for getting started. That is to find a single skill set your initial team can focus on to create fastest returns. We suggested that was Predictive Modeling and laid out four steps for implementation of that Phase 1.

Here in our final Part 2 we show the remaining four phases and lay out an implementation sequence and rationale for following this path.

Phase 2: Geospatial Analytics

geospatial analyticsRemember that we’re being guided by getting the largest, fastest ROI for our efforts. While there are many directions you could go after mastering basic Predictive Analytics, the question is what will provide the biggest return that our current team can provide with the addition of the least additional skills.

The concepts in geospatial analytics (GSA) are very similar to the lift models in predictive analytics but add the variables of location of an event or customer in space, and potentially within time. Some addition to skills or the capabilities of the analytic packages you use will be required but they are not disruptively large. It is possible that your primary ‘model jockey’ will have this skill or you may need to add a specialist or upgrade the qualifications of your team.

The most basic uses of geospatial analysis will warm the heart of any CFO or CMO.

Step 1: Store Performance:

If two stores are producing the same revenue and the same gross margin are they therefore performing the same? Absolutely not. It depends on the accessible customer pool in which they are located. The core concept here is to model the characteristics of the in-store customer and to compare them to the characteristics of the surrounding population. It would not be unusual to find that one store is twice as productive at attracting in-store sales as another clearly indicating unequal performance.

GSA allows you to draw geographic boundaries around store locations which can be as simple as Euclidean shapes or as complex as drive-time boundaries. It is also possible to map your in-store customers onto the map to further refine these boundaries.

There are challenges in data gathering to capture the addresses of in-store customers but there are a number of strategies for doing so. This can be as simple as asking for their address, working from mailed catalogue or flyer lists, or reverse engineering data from credit card sales.

It is likely you will need some specialized software as well as the skills to operate them. These costs are a fraction of one FTE so not budget breaking.

Step 2: Store Location

Similar in concept to step 1, GSA uses the historical patterns of similar stores to look for appropriate concentrations of potential customers around available retail sites. The potential value of a physical location can be directly predicted by this analysis.

There are more sophisticated applications of GSA but they require the use of a NoSQL database and we don’t get to that level of investment until the next step.

Phase 3: NoSQL (Hadoop) Based Analysis of Varied Data Types

hadoop-elephant-smallIt’s not all that expensive to acquire a NoSQL DB and the associated analytic platforms. You can rent the NoSQL DB on AWS or set up your own for a hardware/software cost in the low to mid five figures. But you will need to significantly expand the skills on your team to include DBA and systems architects for NoSQL. If you’re using an analytics platform like SAS, SPSS, or Alteryx, or have good R skills on your team you’re ready to move forward.

The analytic techniques associated with being able to use Big Data (here focusing on a variety of data types) are almost too varied. You know, too many choices is the mother of indecision. We’ll attempt to keep to the game plan and prioritize these by potential payoff.

Step 1: Text Analytics and Natural Language Processing

When most people read these phrases they immediately think of sentiment analysis and monitoring social media. It is true that controlling the customer experience does require monitoring and responding to negative social media and learning from the good things folks may be tweeting. However I think the most powerful application is actually in dynamic pricing.

Using NLP you can design programs to scrape the web for your competitors’ real time competitive pricing information. More practically you can hire specialty firms to do this for you, then store and analyze the output in your NoSQL DB. It is possible to highly automate this whole process but it may be more practical to use some human labor and a definition of ‘real time’ that may be more like overnight or weekly, than down to the nearest minute.

It should be obvious that the ability to respond to competitors’ pricing within a very short time frame can increase sales and market share. If you want to be as sophisticated as Amazon you can even vary pricing based on how and what time of day the customer is accessing your web site. You can also create intentional differences in pricing strategies between the web and in-store, as well as keeping an eye on what your wholesale channel pricing is doing through their independent outlets.

Dynamic pricing can have immediate and substantial economic impact.

Step 2: Click Stream Analytics:

Large numbers of specialized procedures exist for tracking and improving the customers’ experience on your web site, ranging from simple A/B testing through sophisticated presentation of alternate offers based on path. All require the capture and analysis of click stream data. You will need to significantly expand the skills of your team or hire an outside firm to help you with this.

You will need to do a cost-benefit analysis on all these potential projects and any company should certainly commit to a B2C Marketing Automation and Management System as a focus for central control for all these potential activities. (Note that Marketing Automation and Management Systems centralize control over all messages in all channels, not just Content Management of your web site).

While you are working through all those options, I suggest you start here: Recommenders.

recommendersRecommenders are those displays on your web site that suggest what we would like to watch, what else we would like to buy, and even who we might want to date based on our previous browsing and purchasing behavior. This is straightforward cross-sell and upsell and that’s why I would give this priority.

A few years back it was necessary to work with NoSQL Graph DBs to build recommenders and many of the big players today still use Graph DBs. However, these can now be built using simple columnar NoSQL DBs (MapR for example says they can build recommenders in real time during the customer’s web visit) and even companies like SAS provide the capability to build recommenders without resorting to Graph DBs.

Since these lead directly to additional sales and because the technology is relatively straight forward, I suggest you start here.

Step 3: Geophysical Spatial Analysis (GSA):

Using click stream data and other signals from mobile devices it is now possible to establish a geo-fence around your potential customer not only down to the location of your store but down to where they are standing in the aisle and what merchandise is immediately adjacent to their location.

The implication is pretty straightforward. Send the customer a promotional offer relating to merchandise that is within arm’s reach, or at least inside your store to encourage them to act now.

The largest retailers are actively experimenting with this and reporting some pretty positive results. However, we are now reaching the point in our recommendations where the technology is quite cutting edge and may not be appropriate for all. This technique for example requires significant investment in sensors, support, and analytic talent.

Phase 4: Forecasting, Optimization, Simulation

These three analytic techniques have been around for a long time and were born in the discipline of Operations Research. Some pretty sophisticated forecasting can be accomplished with the skills of Predictive Modeling you mastered in Phase 1. Optimization and Simulation use elements of data science combined with mathematics and statistics to select the single best answer to a complex problem.

forecastingForecasting is clearly not eye-balling an extension of the demand line from its current position. Companies like SAS provide very extensive forecasting packages incorporating all the latest thinking and procedures in projecting outcomes and incorporating what-if scenarios.

Optimization is a type of prescriptive analytics that finds a “best” solution from a set of “feasible” solutions, using a mathematical algorithm that maximizes or minimizes a specified objective function subject to constraints. Common approaches to solve this problem include linear programming, integer programming, stochastic programming and constraint programming.

Simulation is a predictive analytics approach that involves building a model to imitate a system or process. The goal is to study the behavior of how it works or run “what if” scenarios around a real-world process. The two most common approaches are “Monte Carlo” and “discrete event” simulations. Simulation and optimization are among the most computationally challenging disciplines of advanced analytics and the skills requirement is substantial.

Your CFO or Chief Supply Chain Officer are likely to be the biggest fans of these techniques. And while they can increase revenue and profit, they are frequently thought of as controlling risk or used for forecasting materials lead times in complex manufacturing and supply chain scenarios.

Phase 5: All the Rest

You could easily spend five or more years mastering Phases 1, 2, and 3, producing a lot of incremental revenue and margin, and making your boss a hero many times over. If you follow the press however, you are bombarded by exotic new developments and capabilities all designed to tempt you to jump in at the bleeding edge. Don’t do it. Follow the outline of good quality, cost effective, and proven blocking and tackling laid out above.

Gartner Research is best known for tracking new and emerging technologies and tracking their progress along their “hype cycle”. The following list is drawn from the most current version of their report for Advanced Analytics and Data Science. These are in no particular order.

Real Time / In Memory / Stream Processing / Internet of Things

I listed these together because they are frequently combined together in very confusing ways in the press. Real Time is valuable for some but does not necessarily require in memory or stream processing. Note that the first project in Phase 1 had you create customized scripts for real time up sell and cross sell guidance to CSRs dealing with customers and no particularly fancy technology was required. As SSD memory becomes increasingly less expensive you may find yourself having access to in-memory analytics in the normal process of IT upgrades, for example what SAP is currently doing by changing out its normal DBs with in-memory HANA DBs. If you have a sensor-heavy environment or the need to intervene in your customer’s behavior while it is happening there could be value here, but in most cases batch analytics combined with click stream detection will accomplish very much the same thing.

Natural Language Generation (NLG) combines natural-language processing (NLP)

In Phase 3 you used NLP as the basis for a dynamic pricing program. You may also have moved into monitoring social media to better control your customer’s experience. What’s on the hype curve however is more exotic.

Whereas NLP is focused on deriving analytic insights from text data, NLG is used to synthesize text, written content by combining analytic output with contextualized narratives. NLG is also being used for operational/regulatory report automation (e.g. threat assessments).

Very early in market adoption and penetration, NLG is already being used to reduce the time and cost to conduct repeatable analysis and writing reports on data. These tend to be for required operational and regulatory report automation, in financial services (earnings reports), government (benefits statements or weather forecasts) and in advertising (personalized messages).

Graph Analysis

Enthusiasts of Graph DBs like to say that they can do anything that row or column based DBs can do plus a lot more. They’re no doubt correct. The challenge is that working with Graph DBs requires a whole new way of thinking about data and will likely require one or more specialists to be added to your team. Plus many of the high payoff opportunities like recommenders and fraud detection can now be accomplished with more traditional NoSQL DB types.

Graph DBs explore relationships between nodes (entities of interest that may be organizations, people, products, or transactions) and focus on the strength and number of connections with other nodes. They are indeed good at things like route optimization, market basket analysis, fraud detection, social network analysis, CRM optimization, supply chain monitoring, and load balancing.

Deep Learning

Although there is a lot of press devoted to deep learning the number of commercial applications is still quite small, mostly in the area of image detection and recognition. Deep learning is a form of unsupervised modeling which is true machine learning, and is based on neural net algorithms. Let this one mature quite a bit before diving in.

Speech Analytics

Audio mining/speech analytics embrace keyword, phonetic or transcription technologies to extract insights from prerecorded and, more recently, real-time voice streams. This insight can then be used to classify calls, trigger alerts/workflows, and drive operational and employee performance across the enterprise. See our previous comments on determining whether real-time analytics is something you really need.

Video Analytics

In its simplest form, video analytics has been used in manufacturing process control for many years to automatically spot and take action on special conditions on the assembly line, like poorly filled bottles in a bottling line. What’s on the hype curve however sf video analytics to evaluate human behavior. This works primarily in real-time video streams to develop context-appropriate automated responses and guide consumer buying behavior.

Emotion Detection/Recognition

Put Audio and Video analytics together and focus them close-up on the human interaction. Detects context sensitive cues in real time streams based on facial expressions, gestures, posture, tone, vocabulary, respiration and skin physiology (temperature and clamminess). Understanding what a person is communicating involves deciphering that individual’s modulation scheme. This requires knowledge of social and cultural mores and cues, as well as a familiarity with the individual. Emotion recognition will lead to optimal computing only if the system is able to consider all of these factors.

Prescriptive Analytics

Simply put, “prescriptive analytics” describes a set of analytical capabilities that specify a preferred course of action. We have been skeptical about breaking this out as a separate capability since it is based on predictive models with the application of some simple optimization mathematics. We’ve been doing that for a long time without resorting to creating a new category. See our earlier article “Prescriptive Analytics”.

Machine Learning

This is another ‘category’ that is very loosely used in the press. We like many others have used ‘machine learning’ to describe the combination of traditional supervised (predictive modeling), unsupervised (clustering), and new NoSQL analytic techniques. Gartner would have us use this phrase in a more tightly controlled way meaning only techniques that are unsupervised (the machine can learn without reference to known examples). This would then encompass very well used clustering and segmentation models along with deep learning, plus the thrust into citizen data science assisted by analytic platforms that operate without the help of expert data scientists.


October 12, 2015

Bill Vorhies, President & Chief Data Scientist – Data-Magnum – © 2015, all rights reserved.

About the author: Bill Vorhies is President & Chief Data Scientist at Data-Magnum and has practiced as a data scientist and commercial predictive modeler since 2001. Bill is also Editorial Director for Data Science Central. He can be reached at: or