researcher who developed it.
National ID numbers in some countries (although not the United States) encode the gender and data of birth of the individual. This is a good and accurate source of this demographic information, when it is available.
Names
Although we want to get to know the customers, the goal of data mining is not to actually meet them. In general, names are not a useful source of information for data mining. There are some cases where it might be interesting to classify names according to ethnicity (such as Hispanic names or Asian names) when trying to reach a particular market or by gender for messaging purposes.
However, such efforts are at best very rough approximations and not widely used for modeling purposes.
Addresses
Addresses describe the geography of customers, which is very important for understanding customer behavior. Unfortunately, the post office can understand many different variations on how addresses are written. Fortunately, there are service bureaus and software that can standardize address fields.
470643 c17.qxd 3/8/04 11:29 AM Page 556
556 Chapter 17
One of the most important uses of an address is to understand when two addresses are the same and when they are different. For instance, is the delivery address for a product ordered on the Web the same as the billing address of the credit card? If not, there is a suggestion that the purchase is a gift (and the suggestion is even stronger if the distance between the two is great and the giver pays for gift wrapping!).
Other than finding exact matches, the entire address itself is not particularly useful; it is better to extract useful information and present it as additional fields. Some useful features are:
■■
Presence or absence of apartment numbers
■■
City
■■
State
■■
Zip code
The last three are typically stored in separate fields. Because geography often plays such an important role in understanding customer behavior, we recommend standardizing address fields and appending useful information such as census block group, multi-unit or single unit building, residential or business address, latitude, longitude, and so on.
Free Text
Free text poses a challenge for data mining, because these fields provide a wealth of information, often readily understood by human beings, but not by automated algorithms. We have found that the best approach is to extract features from the text intelligently, rather than presenting the entire text fields to the computer.
Text can come from many sources, such as:
■■
Doctors’ annotations on patient visits
■■
Memos typed in by call-center personnel
■■
Email sent to customer service centers
■■
Comments typed into forms, whether Web forms or insurance forms
■■
Voice recognition algorithms at call centers
Sources of text in the business world have the property that they are ungrammatical and filled with misspellings and abbreviations. Human beings generally understand them, but it is very difficult to automate this understanding. Hence, it is quite difficult to write software that automatically filters spam even though people readily recognize spam.
470643 c17.qxd 3/8/04 11:29 AM Page 557
Preparing Data for Mining 557
Our recommended approach is to look for specific features by looking for specific substrings. For instance, once upon a time, a Jewish group was boycotting a company because of the company’s position on Israel. Memo fields typed in by call-center service reps were the best source of information on why customers were stopping. Unfortunately, these fields did not uniformly say
“Cancelled due to Israel policy.” In fact, many of the comments contained references to “Isreal,” “Is rael,” “Palistine” [sic], and so on. Classifying the text memos required looking for specific features in the text (in this case, the presence of “Israel,” “Isreal,” and “Is rael” were all used) and then analyzing the result.
Binary Data (Audio, Image, Etc.)
Not surprisingly, there are other types of data that do not fall into these nice categories. Audio and images are becoming increasingly common. And data mining tools do not generally support them.
Because these types of data can contain a wealth of information, what can be done with them? The answer is to extract features into derived variables.
However, such feature extraction is very specific to the data being used and is outside the scope of this book.
Data for Data Mining
Data mining expects data to be in a particular format:
■■
All data should be in a single table.
■■
Each row should correspond to an entity, such as a customer, that is relevant to the business.
■■
Columns with a single value should be ignored.
■■
Columns with a different value for every column should be ignored—
although their information may be included in derived columns.
■■
For predictive modeling, the target column should be identified and all synonymous columns removed.
Alas, this is not how data is found in the real world. In the real world, data comes from source systems, which may store each field in a particular way.
Often, we want to replace fields with values stored in reference tables, or to extract features from more complicated data types. The next section talks about putting this data together into a customer signature.
470643 c17.qxd 3/8/04 11:29 AM Page 558
558
Chapter 17
Constructing the Customer Signature
Building the customer signature, especially the first time, is a very incremental process. At a minimum, customer signatures need to be built at least two times—once for building the model and once for scoring it. In practice, exploring data and building models suggests new variables and transformations, so the process is repeated many times. Having a repeatable process simplifies the data mining work.
The first step in the process, shown in Figure 17.7, is to identify the available sources of data. After all, the customer signature is a summary, at the customer level, of what is known about each customer. The summary is based on available data. This data may reside in a data warehouse. It might equally well reside in operational systems and some might be provided by outside vendors. When doing predictive modeling, it is particularly important to identify where the target variable is coming from.
The second step is identifying the customer. In some cases, the customer is at the account level. In others, the customer is at the individual or household level. In some cases, the signature may have nothing to do with a person at all.
We have used signatures for understanding products, zip codes, and counties, for instance, although the most common use is for accounts and households.
Identify a working
definition of customer.
Copy most recent
input data snapshot
of customer.
Pivot to produce
multiple months of data
for some data elements.
Calculate churn flag
for the prediction period.
Revisit the customer
definition.
Incorporate other
data sources.
Add derived variables.
Figure 17.7 Building customer signatures is an iterative process; start small and work through the process step-by-step, as in this example for building a customer signature for churn prediction.
470643 c17.qxd 3/8/04 11:29 AM Page 559
Preparing Data for Mining 559
Once the customer has been identified, data sources need to be mapped to the customer level. This may require additional lookup tables—for instance, to convert accounts into households. It may not be possible to find the customers in the available data. Such a situation requires revisiting the customer definition.
The key to building customer signatures is to start simple and build up. Prioritize the data sources by the ease with which they map to the customer. Start with the easiest one, and build the signature using it. You can use a signature before all the data is put into it. While awaiting more complicated data transformations, get your feet wet and understand what is available. When building customer signatures out of transactions, be sure to get all the transactions associated with a particular customer.
Cataloging the Data
The data mining group at a mobile telecommunications company wants to develop a churn model in-house. This churn model will predict churn for one month, given a one-month lag time. So, if the data is available for February, then the churn prediction is for April. Such a model provides time for gathering the data and scoring new customers, since the February data is available sometime in March.
At this company, there are several potential sources of data for the customer signatures. All of these are kept in a data repository with 18 months of history.
Each file is an end-of-the-month snapshot—basically a dump of an operational system into a data repository.
The UNIT_MASTER file contains a description of every telephone number in service and a snapshot of what is known about the telephone number at the end of the month. Examples of fields in this file are the telephone number, billing account, billing plan, handset model, last billed date, and last payment.
The TRANS_MASTER file contains every transaction that occurs on a particular telephone number during the course of the month. These are account-level transactions, which include connections, disconnections, handset upgrades, and so on.