A good way to attract attention and budget dollars to a project is to use data mining to meet a real business need. The most convincing proof-of-concept projects focus on areas that are already being measured and evaluated analytically, and where there is already an acknowledged need for improvement.
Likely candidates include:
■■
Response models
■■
Default risk models
■■
Attrition models
■■
Usage models
■■
Profitability models
470643 c18.qxd 3/8/04 11:31 AM Page 600
600 Chapter 18
These are areas where there is a well-defined link between improved accuracy of predictions and improved profitability. With some projects, it is easy to act on the data mining results. This is not to say that pilot projects with a focus on increased insight and understanding without any direct link to the bottom line cannot be successful. They are, however, harder to build a business case for.
Potential users of new information are often creative and have good imaginations. During interviews, encourage them to imagine ways to develop true learning relationships with customers. At the same time, make an inventory of available data sources, identifying additional fields that may be desirable or required. Where data is already being warehoused, study the data dictionaries and database schemas. When the source systems are operational systems, study the record layouts that will be supplying the data and get to know the people who are familiar with how the systems process and store information.
As part of the proof-of-concept selection process, do some initial profiling of the available records and fields to get a preliminary understanding of relationships in the data and to get some early warnings of data problems that may hinder the data mining process. This effort is likely to require some amount of data cleansing, filtering, and transformation.
Once several candidate projects have been identified, evaluate them in terms of the ability to act on the results, the usefulness of the potential results, the availability of data, and the level of technical effort. One of the most important questions to ask about each candidate project is “how will the results be used?” As illustrated by the example in the sidebar “A Successful Proof of Concept?” a common fate of data mining pilot projects is to be technically successful but underappreciated because no one can figure out what to do with the results.
There are certainly many examples of successful data mining projects that originated in IT. Nevertheless, when the people conducting the data mining are not located in marketing or some other group that communicates directly with customers, sponsorship or at least input from such a group is important for a successful project. Although data mining requires interaction with databases and analytic software, it is not primarily an IT project and should rarely be attempted in isolation from the owners of the business problem being addressed.
T I P A data mining pilot project may be based in any of several groups within the company, but it must always include active participation from the group that feels ownership of the business problem to be addressed.
Marketing campaigns make good proof-of-concept projects because in most companies there is already a culture of measuring the results of such campaigns. A controlled experiment showing a statistically significant improvement in response to a direct mail, telemarketing, or email campaign is easily translated into dollars. The best way to prove the value of data mining is with
470643 c18.qxd 3/8/04 11:31 AM Page 601
Putting Data Mining to Work 601
A SUCCESSFUL PROOF OF CONCEPT?
A data mining proof of concept project can be technically successful, yet disappointing overall. In one example, a cellular telephone company launched a data mining project to gain a better understanding of customer churn. The project succeeded in identifying several customer segments with high churn risk. With the groups identified, the company could offer these customers incentives to stay. So far, the project seems like a good proof-of-concept that returns actionable results.
The data mining models found one group of high-risk customers, consisting of subscribers whose calling behavior did not match their rate plans. One subgroup of these customers were on rate plans with low monthly fees, and correspondingly few included minutes. Such plans make sense for people who use their phones infrequently, such as the “safety user” who leaves a telephone in the car’s glove compartment, rarely turning it on but more secure in the knowledge that the phone is available for emergencies. When such users change their telephone habits (as sometimes happens once they realize the usefulness of a mobile phone), they end up using more minutes than are included in their plan, paying high per minute charges for the overage.
The company declared the data mining project a success because the groups that the model identified as high risk were tracked and did in fact leave in droves. However, nothing was done because the charter of the group sponsoring the data mining project was to explore new technologies rather than manage customer relationships. In a narrow sense, the project was indeed successful. It proved the concept that data mining could identify customers at high risk for churn.In a broader sense, the organization was not ready for data mining, so it could not successfully act on the results.
There is another organizational challenge with these customers. As long as they remain, the mismatched customers are quite profitable, paying for expensive overcalls or on a too-expensive rate plan. Moving them to a rate plan that saved them money (“right-planning” them) might very well decrease churn but also decrease profitability. Which is more important, churn or profitability?
Data mining often raises as many questions as it answers, and the answers to some questions depend on business strategy more than on data mining results.
a demonstration project that goes beyond evaluating models to actually measuring the results of a campaign based on the models. Where that is not possible, careful thought must be given to how to attach a dollar value to the results of the demonstration project. In some cases, it is sufficient to test the new models derived from data mining against historical data.
Implementing the Proof-of-Concept Project
Once an appropriate business problem has been selected, the next step is to identify and collect data that can be transformed into actionable information.
Data sources have already been identified as part of the process of selecting the
470643 c18.qxd 3/8/04 11:31 AM Page 602
602 Chapter 18
proof-of-concept project. The next step is to extract data from those sources and transform it into customer signatures, as described in the previous chapter. Designing a good customer signature is tricky the first few times. This is an area where the help of experienced data miners can be valuable.
In addition to constructing the initial customer signature, there needs to be a prototype data exploration and model development environment. This environment could be provided by a software company or data mining consultancy, or it can be constructed in-house as part of the pilot project. The data mining environment is likely to consist of a data mining software suite installed on a dedicated analytic workstation. The model development environment should be rich enough to allow the testing of a variety of data mining techniques. Chapter 16 has advice on selecting data mining software and setting up a data mining environment. One of the goals of the proof-of-concept project is to determine which techniques are most effective in addressing the particular business problem being tackled.
Using the prototype data mining system involves a process of refining the data extraction requirements and interfaces between the environment and the existing operational and decision-support computing environments. Expect this to be an iterative process that leads to a better understanding of what is needed for the future data mining environment. Early data mining results will suggest new modeling approaches and refinements to the customer signature.
TEAMFLY
When the prototype data mining environment has been built, use it to build predictive models to perform the initial high-payback task identified when the proof-of-concept project was defined. Carefully measure the performance of the models on historical data.
It is entirely feasible to accomplish the entire proof-of-concept project without actually building a prototype data mining environment in-house by using external facilities. There are advantages and disadvantages to this approach.
On the positive side, a data mining consultancy brings insights gained through experience working with data from other companies to the problem at hand. It is unlikely that anyone on your own staff has the knowledge and experience with the broad range of data mining tools and techniques that specialists can bring to bear. On the negative side, you and your staff will not learn as much about the data mining process if consultants do all the actual data mining work. Perhaps the best compromise is to put together a team that includes outside consultants along with people from the company.
Act on Your Findings
The next step is to measure the results of modeling. In some case, this is best done using historical data (preferably an out-of-time sample for a good comparison).