Generating Organizational Analytics

Data resides in many places within large companies. One division of a company could sell clothing while another sells sporting goods—and each capture customer data differently. Combining the datasets of these two divisions could help the organization as a whole understand their customers better. This can improve both large-scale planning (what will a lot of people want to buy next season?) and small-scale marketing (what does Andy want for his exercise routine?).

As a simple example, if customers are spending more on sporting goods than expected, they may be open to spending more on clothing as well, especially what they wear during their sporting events. A more sophisticated example comes from sourcing products: analytics may reveal that two functionally interchangeable products cost different amounts, and might lead a company to save money by substituting a cheaper material for a more expensive one.

One Fortune 100 manufacturer uses analytics on its master data of suppliers and parts to control supplier costs, such as by finding the lowest-cost supplier for each part. Using machine learning to master and classify tens of millions of records over a relatively short period of time saved the company hundreds of millions of dollars. On another set of multiple CRM systems, they reduced more than one billion customer records to derive a few hundred thousand master records.

Data Mining

Datasets can also be used to create a mosaic effect, which refers to laying out a wide variety of data elements that say relatively little in isolation but reveal a pattern when combined. For instance, researchers have shown that combining de-identified data from different sets can turn up relationships that re-identify a person.

A legitimate use of the mosaic effect is in law enforcement. Homeland Security may notice that a resident of Los Angeles named Sam Smith was recently fired from his job for inappropriate behavior, and that a resident of Long Beach named “Spiffy” Smith recently bought a semi-automatic weapon. Are Sam Smith and “Spiffy” Smith the same person, and has his behavior changed recently in suspicious ways? Analyzing information from data sets that have been combined and mastered may yield the answer.

Creating Classifications

A classification, or taxonomy, is a way of understanding the world by grouping and categorizing. Should I consider a watch that tracks my footsteps to be an item of clothing, a piece of sporting equipment, or jewelry? It can be multiple things, but I should be able to find it by searching a website in ways that are intuitive to me.

Most classifications keep things simple and arrange products in hierarchies. But as the watch example shows, different taxonomies can classify the same entity in different ways. Master records with different classifications must be harmonized in order to produce a consistent classification.

The classification problem becomes even harder for an organization in the event of a merger or acquisition. Naturally, classifications that come from different companies during a merger or acquisition are likely going to be very different. The organization must start by deciding which classification to adopt. All the records of the other company must be fit into the chosen classification, although the chosen one can be expanded. But the organization must determine how the thousands or millions of new records match to the chosen classification.

The Advantages of Agile Data Mastering

In this section, we look at processes based on machine learning that can create master data records and related outputs, such as classifications. This section is based on the operation of Tamr, a relatively young company that applies human-guided machine learning to data mastering.

Traditional techniques of determining if records match has always involved the application of rules. As we have seen, current MDM usually requires people to define the rules and embed them in programs. One rule, for instance, might accept a match if the text in two different records is sufficiently similar, while another might reject a match if two birthdates are sufficiently far apart in time. Rules can also involve sophisticated processing, such as setting the weight of different fields. Statisticians may determine that matching a name is much more important than matching certain other fields that are more likely to change, be entered incorrectly, or be less important. So the name fields will have a higher weight when trying to determine whether two records refer to the same entity. Similar techniques for matching records are used for merging them into master data records and for classifying them into taxonomies.

With agile data mastering, finding the same words in different records (one classified and another not classified) allows Tamr to determine whether records are referring to the same entity and/or where they should be placed in a taxonomy. And if matches are not exact, machine learning can determine that the fields are close enough to classify new records properly.

Tamr does not start out with rules; it finds all the relationships, weights, and probabilities through machine learning. There are many advantages to generating rules dynamically:

• The system adapts to the data presented by each organization, generating record matches that are uniquely appropriate for the domain of interest (customers, suppliers, products, etc.). This is important for organizations in different industries or areas of research, with arcane terminology and relationships.

• The resulting rules are more complex than a human being could design or understand, allowing for more accuracy in matching and classification.

• The system is fast, generating results in a few hours.

• Cultural and language differences are handled automatically.

• The system can be run regularly to pick up changes in data, keeping the rules in sync with the environment.

Machine learning is essentially the balancing of probabilities. Thus, in the case of Pei Jin-Li, the two records in Table 1-2 are very similar and therefore are highly likely to refer to the same person.

On the other hand, the records in Table 1-3 are less similar, and less likely to be the same person.

Because it involves probabilities, machine learning also involves risk. Algorithms might calculate, based on all the records it finds about Pei in Chicago, that the previous two records have a 90% chance of being the same person. If the algorithm accepts 1,000 matches with a 90% probability of being right, your master data will contain 100 people who are wrongly associated with another person (100 false positives, or type I errors). But this may be better than rejecting those one thousand matches, because then you’ll have 900 duplicates (false negatives, or type II errors). There is always some balance between false positives and false negatives; decreasing one risk leads to an increase in the other. In the field of information retrieval, this balance can also be viewed as a balance between precision (did we get each match right?) and recall (did we get all available matches?).

Many machine learning algorithms tell you the risk of false positives and negatives, and allow you to tune the algorithm. A false positive for customers, as we’ve seen, may lead you to market a product to Jin-Li that she’s not interested in, which is not a particularly bad outcome. In contrast, false positives may have much more dangerous consequences if you are matching parts from vendors. You might end up placing 10,000 orders for a part that you think performs the same role as a current part, but is cheaper. You may then discover that the cheap part differs from the current part in some crucial way that renders it unfit for your use.

Part of the implementation of an algorithm, therefore, involves handling ambiguous cases where there is a risk of a false positive or negative. We’ll see how Tamr handles these cases as we look at its operation.

Goals of Training

We all learn from experience, what logicians call inductive reasoning. This kind of learning—deriving general principles from a large set of specific examples—also characterizes the training that is a critical part of machine learning. The process starts with data that has been checked by humans and is guaranteed to be accurate; after processing this data, the machine learning can start to understand record matching principles that help it process new, raw data.

Below are examples of intelligence that machine learning systems can develop by running statistics on large amounts of data:

Outliers

These are entries that differ wildly from the others in some way. For instance, if data indicates that a doctor performed 200 surgeries in one day, machine learning will probably flag him as someone to examine. The reason may be an error in programming or data entry. In other circumstances, it can uncover fraud. Or it may simply occur because one doctor is billing for all the surgeries performed in his department.

Outliers can be recognized without applying explicit rules such as, “A doctor can perform at most three surgeries each day.” Instead, the outliers are discovered during clustering. The outlier may have some features (data fields) in common with other records, but at least one of the features is so different that it ends up far away from the other records. If one were to graph the data, one might see several groups of records that are related, and a few outliers that lie off in their own regions of the graph with large gaps between them and the rest of the data.

Typos and other inaccuracies

These are particular types of outliers. Unlike the doctor performing 200 surgeries, a typo is not wildly different from other values. Instead, it is only slightly different: “Avnue” instead of “Avenue.” Machine learning can identify “Avnue” as an outlier because it occurs rarely, is very similar to the correct spelling, and appears in a context that makes the similarity clear—for instance, the rest of the address is the same as in the fields that say “Avenue.”

Ranges and limits

These identify other kinds of outliers. Machine learning will discover that an age of -1 or 200 for a person is an error.

Patterns and arbitrary rules

For instance, a state called Cal is probably California, and Bob is an alternative name for Robert. As with typos, these variants can be discovered through context. As we saw with Pei Jin-Li, machine learning can compare several similar records to suggest that Julie is an alternative for Jin-Li. The match would be an impossible stretch if we saw the two names in isolation, but in the context of the other fields, the match seems probably correct.

The Training Process

Tamr has a unique training process that involves human input. Initial training usually requires a few hours.

The process starts with known matches, provided by the customer, and derives as much information as it can from the input data on its own. It can compare most records quickly, because most records are very different. When Tamr traverses columns from different data sets and compares fields, it determines how close two strings are through common algorithms such as Levenshtein distance. (That’s just one classic formula for checking whether strings are similar; there are many others.) The word “Avenue” is close to the misspelled “Avnue,” creating what is called a strong signal in machine learning, whereas Avenue is less close to Ave. As we saw in the Pei Jin-Li example, machine learning can determine that Avenue and Ave refer to the same thing by comparing large numbers of records where other strings within the field, or other fields, are strongly matched.

Certain fields will predict matches better than other fields, so Tamr can assign a weight to each field during training to determine how much confidence to invest in that field later when processing raw data. The result of applying traditional clustering techniques is a huge set of very small clusters. For instance, the Pei Jin-Li shown in Table 1-1 contained only five records. There could be tens of thousands of these clusters, one for each customer.

If confidence is high that a row belongs in a cluster, it is associated with the master data record and may supply some of the data values for that record. Each cluster results in a single row in the master data table.

Inevitably, some records will be ambiguous. They are so similar that they might be considered a match, yet different enough to be suspect. Machine learning can make confident decisions on many records by comparing all the fields of the records, assigning more confidence to higher-weighted fields, and by finding patterns that appear frequently in the data. For instance, if Bob and Robert have been shown to refer to the same person in a thousand records, the machine learning algorithms place more confidence that Bob equals Robert when comparing two records that differ in other ways as well. When the decision that they’re the same is not clear enough, Tamr can present the records to a human. Having humans mark the pair as correct or incorrect through very simple yes or no questions helps Tamr tune its algorithms.

To improve its classification process, Tamr follows a similar process: using its algorithms to attempt to match records to a taxonomy, and then asking a human user a simple yes/no question for validation if necessary. For instance, Tamr may start with records associated with a taxonomy that contains branches for nuts and bolts. Records from a recently acquired company are then input, some describing half-inch steel screws, some describing quarter-inch copper screws, and so on. Because the original taxonomy contains bolts that are half-inch steel or quarter-inch copper, Tamr can recognize that the screws are the same as bolts—but it will ask for human intervention if it is not sure.

Thus, the user helps to train the algorithm with just a few simple choices. With each iteration of training the algorithm, it can classify more and more records with less and less human help. One media company achieved both precision and recall rates of over 95% with Tamr.

Machine learning requires retraining over time. Tamr will continue to refine both the match/merge algorithm and the classification algorithm for an organization’s data by processing new data on a regular basis.