102 Chapter 4
BENEFIT (continued)
As used in the medical world, sensitivity is the proportion of true positives among people who get a positive result on a test. In other words, it is the true positives divided by the sum of the true positives and false positives. Sensitivity measures the likelihood that a diagnosis based on the t
Specificity
est is correct.
is the proportion of true negatives among
people who get a negative result on the test. A good test should be both sensitive and specific. The maximum benefit point is the cutoff that maximizes the average of these two measures. In Chapter 8, these concepts go by the names recall and precision, the terminology used in information retrieval. Recall measures the number of articles on the correct topic returned by a Web search or other text query. Precision measures the percentage of the returned articles that are on the correct topic.
◆ The maximum benefit point corresponds to a decision rule that minimizes the expected loss assuming the misclassification costs are inversely proportional to the prevalence of the target classes.
One way of evaluating classification rules is to assign a cost to each type of misclassification and compare rules based on that cost. Whether they represent responders, defaulters, fraudsters, or people with a particular disease, the rare cases are generally the most interesting so missing one of them is more costly than misclassifying one of the common cases. Under TEAMFLY
that assumption, the maximum benefit picks a good classification rule.
This table says that if a prospect is contacted and responds, the company makes forty-four dollars. If a prospect is contacted, but fails to respond, the company loses $1. In this simplified example, there is neither cost nor benefit in choosing not to contact a prospect. A more sophisticated analysis might take into account the fact that there is an opportunity cost to not contacting a prospect who would have responded, that even a nonresponder may become a better prospect as a result of the contact through increased brand awareness, and that responders may have a higher lifetime value than indicated by the single purchase. Apart from those complications, this simple profit and loss matrix can be used to translate the response to a campaign into a profit figure.
Ignoring campaign overhead fixed costs, if one prospect responds for every 44
who fail to respond, the campaign breaks even. If the response rate is better than that, the campaign is profitable.
WA R N I N G If the cost of a failed contact is set too low, the profit and loss matrix suggests contacting everyone. This may not be a good idea for other reasons. It could lead to prospects being bombarded with innapropriate offers.
Team-Fly®
470643 c04.qxd 3/8/04 11:10 AM Page 103
Data Mining Applications 103
How the Model Affects Profitability
How does the model whose lift and benefit are characterized by Figure 4.2
affect the profitability of a campaign? The answer depends on the start-up cost for the campaign, the underlying prevalence of responders in the population and on the cutoff penetration of people contacted. Recall that SAC had a budget of $300,000. Assume that the underlying prevalence of responders in the population is 1 percent. The budget is enough to contact 300,000 prospects, or 30 percent of the prospect pool. At a depth of 30 percent, the model provides lift of about 2, so SAC can expect twice as many responders as they would have without the model. In this case, twice as many means 2 percent instead of 1 percent, yielding 6,000 (2% * 300,000) responders each of whom is worth $44 in net revenue. Under these assumptions, SAC grosses $600,000 and nets $264,000
from responders. Meanwhile, 98 percent of prospects or 294,000 do not respond. Each of these costs a dollar, so SAC loses $30,000 on the campaign.
Table 4.4 shows the data used to generate the concentration chart in Figure 4.2. It suggests that the campaign could be made profitable by spending less money to contact fewer prospects while getting a better response rate. Mailing to only 10,000 prospects, or the top 10 percent of the prospect list, achieves a lift of 3. This turns the underlying response rate of 1 percent into a response rate of 3 percent. In this scenario, 3,000 people respond yielding revenue of $132,000. There are now 97,000 people who fail to respond and each of them costs one dollar. The resulting profit is $35,000. Better still, SAC has $200,000
left in the marketing budget to use on another campaign or to improve the offer made in this one, perhaps increasing response still more.
Table 4.4 Lift and Cumulative Gains by Decile
CUMULATIVE
PENETRATION
GAINS
GAINS
LIFT
0%
0%
0%
0
10%
30%
30%
3.000
20%
20%
50%
2.500
30%
15%
65%
2.167
40%
13%
78%
1.950
50%
7%
85%
1.700
60%
5%
90%
1.500
70%
4%
94%
1.343
80%
4%
96%
1.225
90%
2%
100%
1.111
100%
0%
100%
1.000
470643 c04.qxd 3/8/04 11:10 AM Page 104
104 Chapter 4
A smaller, better-targeted campaign can be more profitable than a larger and more expensive one. Lift increases as the list gets smaller, so is smaller always better? The answer is no because the absolute revenue decreases as the number of responders decreases. As an extreme example, assume the model can generate lift of 100 by finding a group with 100 percent response rate when the underlying response rate is 1 percent. That sounds fantastic, but if there are only 10 people in the group, they are still only worth $440. Also, a more realistic example would include some up-front fixed costs. Figure 4.3 shows what happens with the assumption that there is a $20,000 fixed cost for the campaign in addition to the cost of $1 per contact, revenue of $44 per response, and an underlying response rate of 1 percent. The campaign is only profitable for a small range of file penetrations around 10 percent.
Using the model to optimize the profitability of a campaign seems more attractive than simply using it to pick whom to include on a mailing or call list of predetermined size, but the approach is not without pitfalls. For one thing, the results are dependent on the campaign cost, the response rate, and the revenue per responder, none of which are known prior to running the campaign.
In the example, these were known, but in real life, they can only be estimated.
It would only take a small variation in any one of these to turn the campaign in the example above completely unprofitable or to make it profitable over a much larger range of deciles.
Profit by Decile
$100,000
$0
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
($100,000)
($200,000)
($300,000)
($400,000)
($500,000)
($600,000)
Figure 4.3 Campaign profitability as a function of penetration.
470643 c04.qxd 3/8/04 11:10 AM Page 105
Data Mining Applications 105
$400,000
$200,000
base
20% down
20% up
$0
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
($200,000)
($400,000)
($600,000)
($800,000)
($1,000,000)
Figure 4.4 A 20 percent variation in response rate, cost, and revenue per responder has a large effect on the profitability of a campaign.
Figure 4.4 shows what would happen to this campaign if the assumptions on cost, response rate, and revenue were all off by 20 percent. Under the pessimistic scenario, the best that can be achieved is a loss of $20,000. Under the optimistic scenario, the campaign achieves maximum profitability of $161,696
at 40 percent penetration. Estimates of cost tend to be fairly accurate since they are based on postage rates, printing charges, and other factors that can be determined in advance. Estimates of response rates and revenues are usually little more than guesses. So, while optimizing a campaign for profitability sounds appealing, it is unlikely to be possible in practice without conducting an actual test campaign. Modeling campaign profitability in advance is primarily a what-if analysis to determine likely profitability bounds based on various assumptions. Although optimizing a campaign in advance is not particularly useful, it can be useful to measure the results of a campaign after it has been run. However, to do this effectively, there need to be customers included in the campaign with a full range of response scores—even customers from lower deciles.
WA R N I N G The profitability of a campaign depends on so many factors that can only be estimated in advance that the only reliable way to do it is to use an actual market test.
470643 c04.qxd 3/8/04 11:10 AM Page 106
106 Chapter 4
Reaching the People Most Influenced by the Message
One of the more subtle simplifying assumptions made so far is that when a model with good lift is identifying people who respond to the offer. Since these people receive an offer and proceed to make purchases at a higher rate than other people, the assumption seems to be confirmed. There is another possibility, however: The model could simply be identifying people who are likely to buy the product with or without the offer.