Structure

The structure of a dataset refers to the format and encoding of its records and fields. We can consider datasets on a spectrum related to the homogeneity of their records and fields. At one end of the spectrum, the dataset is “rectangular” and can be formatted as a table with a fixed number of rows and columns. In this format, rows in the table correspond to records, and columns correspond to fields.

If the record fields in a dataset are not consistent (some records have additional fields, others are missing fields, etc.), you could be dealing with a “jagged” table. Such a table is no longer perfectly rectangular. Data formats like JSON and XML support datasets like this, in which record fields are not fully consistent.

Further along the spectrum are datasets with a heterogeneous set of records. For example, a heterogeneous dataset from a retail organization might mix customer information and customer transactions. This is a common occurrence if you consider the tabs in a complex Excel workbook. Most analysis and visualization tools will require these different kinds of records to be split into separate files or tables. Analyses that require all of these records will often start by joining or blending the records together based on shared fields.

The encoding of a dataset specifies how the record fields are stored and presented to the user. Are the timestamps in local time zones or mapped to UTC? Do date/time fields conform to specifications like ISO 8601? Are ages stored in years, days, or fractions of decades? Are account balances in US dollars? Are the postal codes up to date? Are accented characters supported, or have they been replaced with their nonaccented counterparts? Are right-to-left writing systems, like Arabic, supported?

In many cases, it is advisable to encode a dataset in plain text. This makes the file human-readable. A major drawback to encoding the file in plain text is the size of the file; it is far more space efficient to use binary encodings of numerical values, or to compress the file using standard algorithms like gzip and bzip.

Assessing the structure of your dataset is primarily a generic metadata question. Before you can begin wrangling your data, you need to understand how that data is structured. This often requires counting the number of records and fields in a dataset, and determining the dataset’s encoding.

Beyond these generic metadata concerns, you might need to generate custom metadata pertaining to the specific structure of your dataset. Earlier in this chapter, we discussed CPG organizations that need to work with data provided by their external trading partners. Because this data originates outside of the organization, CPG analysts often need to determine the custom fields that each trading partner is adding to the dataset. Maybe Walmart’s datasets include a slightly different set of fields than those for Target. When you make these determinations, you generate custom metadata.

Granularity

The granularity of a dataset refers to the kinds of entities that each data record represents or contains information about. In their most common form, records in a dataset will contain information about many instances of the same kind of entity.

We typically describe granularity in terms of coarseness and fineness. In the context of data, this means the level of depth or the number of distinct entities represented by a single record of your dataset. For example, a dataset in which a single record represents a single sales transaction by a single customer at a particular store would have a fine granularity. A dataset in which each record represents the total sales in each store for each day would have a coarse granularity. At an even coarser granularity, you might have a dataset in which each record represents total sales by region and week. Depending on your intended use of the dataset, the granularity might be just right, too coarse, or too fine.

There are subtleties to assessing the granularity of a dataset that involve applying organizational knowledge. These are examples of creating custom metadata related to granularity. For example, at first glance, a dataset might appear to contain records that represent customers. However, those records might in fact correspond to all known contacts of your company (only a subset of whom are actual, paying customers). Moreover, this contacts dataset might contain multiple entries for the same person, resulting from that person signing up to receive information through multiple channels (e.g., Facebook and via a direct visit to your website). In this case, a more appropriate description of the granularity of the dataset might be “registration events.”

Accuracy

The accuracy of a dataset refers to its quality. In other words, the values populating record fields in the dataset should be consistent and accurate. For example, consider a customer actions dataset. This dataset contains records corresponding to when customers added items to their shopping carts. In some cases, the reference to the item added to the cart in the record is not accurate; perhaps a UPC code is used but some of the digits are missing, or the UPC code is out of date and has since been reused for a different item. Inaccuracies would, of course, render any analysis on the dataset problematic.

Other common inaccuracies are misspellings of categorical variables, like street or company names; lack of appropriate categories, like ethnicity labels for multiethnic people; underflow and overflow of numerical values; and missing field components, like a timestamp encoded in a 12-hour format but missing an AM/PM indication.

Accuracy assessments can also apply to frequency outliers—values that occur more or less often than you’d expect. Making frequency assessments is primarily a custom metadata concern, given that determining whether the range of values in a dataset is accurate relies on organizational knowledge. Returning to our CPG example, a supply-chain analyst might know that a certain trading partner only reports UPCs within a particular range. In this case, the analyst needs to generate custom metadata around the accuracy of the UPC code distribution in a dataset.

Temporality

A data record is a representation of an entity at a particular time (or set of times). Accordingly, even though a dataset might have been an accurate and consistent representation at the time it was created, subsequent changes might render the representation inaccurate or inconsistent. For example, you might use a dataset of customer actions to determine the distribution of items people own. Weeks or months after an initial sale, however, some of these items might be returned. Now the original dataset, although an accurate representation of the original sales transaction, is no longer an accurate representation of the items a person owns.

The time-sensitive nature of representations, and hence of datasets, is an important aspect that should be explicitly noted. Sometimes this is done on a per-record basis (e.g., each customer action contains a timestamp). Data records that do not correspond to events are less likely to contain embedded timestamps; for example, a customer database with names, addresses, demographics, or a reference dataset of UPCs with item descriptions. However, even when time is not explicitly represented in a dataset, it is still important to understand how time may have impacted the records in the dataset. In the first example, a customer might move and his address can become inaccurate. In the second example, UPC numbers might be recycled and item descriptions can become inaccurate. In all such cases, knowing when the dataset was generated can provide valuable insight into potential inaccuracies and the appropriate wrangling actions needed to remediate those inaccuracies.

Scope

The scope of a dataset has two major dimensions. The first dimension concerns the number of distinct attributes represented in a dataset. For example, for each customer action, we might know when it happened (e.g., a timestamp) and some details about it (like which UPC a customer added to a basket). The second dimension concerns the attribute-by-attribute population coverage: are “all” the attributes for each field represented in the dataset, or have some been randomly, intentionally, or systematically excluded?

Let’s begin our discussion of the importance of scope by addressing the number of distinct attributes in a dataset. Each distinct attribute is generally represented in a dataset by a single field. A dataset with a wide scope will have a large number of fields, whereas a dataset with a narrow scope will have a small number of fields. You could imagine that a customer information dataset with a very wide scope could have hundreds of distinct fields, each representing a different attribute of a customer (age, salary, ethnicity, family size, etc.).

Obviously, increasing the scope of a dataset by adding in more entity characteristics extends the analytical potential of the dataset. However, as with granularity, you want to include only as much detail as you might use, but no more. The level of detail required might depend on your analytics methodology. Some methodologies, like deep learning, call for keeping many redundant attributes and letting statistical methods boil those many attributes down to a smaller number. Other methodologies operate best with a limited number of attributes.

Regarding population coverage, the most common scenario is that not all of the possible entities have been represented. Sometimes, missing records are the result of external factors, like the logging infrastructure failing due to a power outage. Other times you’re missing some of the data due to an operational or logistical error: a chunk of a file was destroyed, a sensor was knocked off power, somebody didn’t give you an entire file, or somebody redacted the dataset. The cause of the missing data can be helpful to know because it can help account for the bias downstream. For example, if you know that a dataset was scrubbed of people under 18 for legal reasons, you should account for the average age in the dataset being higher than the true population that the dataset (imperfectly) represents.

More generally, we might want to introduce the notion of an idea or “true” source of data in the real world, and an imperfect record of that data that was acquired. When statisticians or scientists speak of “sampling,” they usually are talking about this—acquiring a small but representative subset of the “true” data as a stored dataset. The ideal (true) source of data might be infeasible to capture (think about the temperature of every cubic inch of air at every second hovering over the globe—we sample that to get temperature readings at particular thermometers). The data that is recorded is inherently a sample. This is a big difference between real-world (e.g., IoT) data and electronic data like transactions or web click logs.

It is important to understand any systematic bias in a dataset, because for cases in which systematic bias exists, any analytical inferences made using the biased dataset can be rendered invalid. A canonical example is drug trial analysis, where analysts are concerned with assessing the efficacy of the drug being trialed. The granularity of drug trial datasets is often at the level of patients in the trial. If, however, the scope of the dataset has been intentionally reduced by systematically removing records associated with some patients in the trial (either because they died during the trial or because they began showing abnormal biometrics), there’s a good chance that any analysis on the dataset is likely to misrepresent the actual impact of the drug.

The most common custom metadata question is: “How can this dataset blend with (join or union to) my other datasets?” To answer this question, we must understand the scope of the datasets. In some cases, the new dataset might represent an extension of an existing dataset. Sometimes, this extension is disjoint, involving records representing entities in the overall population that are missing in your existing datasets. Sometimes, the dataset extension is overlapping, creating a need to deduplicate or harmonize records representing the same entity. In other cases, a new dataset might provide additional record fields (e.g., household disposable income to be matched against your customer list or counts of different kinds of accidents by postal code).

Addressing structural issues

Most visualization and analysis tools expect tabular data, meaning that every record has the same fields instantiated in the same order. Depending on the structure of the raw data, converting data into tabular format can require significant transformations. Furthermore, for modeling and forecasting purposes, you might need to convert categorical data to separate indicator values for category value; for example, you might need to expand a record field encoding gender into multiple fields corresponding to characteristics like “is male” or “is female.”

Addressing granularity issues

From a granularity perspective, it is prudent to build refined datasets at the finest resolution of records that you plan to analyze. For example, let’s assume that your sales records show a significant increase in the average amount of a sales transaction, and that you can identify a subset of users who form the driver of this shift (this group makes significantly larger purchases relative to other customers, and removing them from the average calculation produces an average sale amount consistent with the recent past). You might want to understand what exactly differentiates these customers from the rest: Are they buying more expensive items? More items than average? Do they shop more often? Do they seem unaware of sales and coupons? Chances are, answering these questions will require the records of actions leading up to the sales transaction; in this case, the records with the finest granularity are most ideal for your analysis.

However, if the majority of the analyses focus on records of a coarser granularity, (e.g., customer segments or demographic groups, and lifetime purchase totals), it also might make sense to store a version of the dataset at this granularity, as well. Retaining multiple versions of the same dataset with different granularities can help streamline downstream analyses based on groups of records.

Addressing accuracy issues

Another key objective in designing and creating refined datasets is to remedy known accuracy issues.

The main strategies used to handle inaccuracies are to: (a) remove records with inaccurate values (provided the inaccuracies can be detected); (b) retain records with inaccurate values but mark them as inaccurate (which still allows some analyses to be conducted over the dataset); or (c) replace the inaccurate values with default or estimated values in a process known as imputation. As an example, for numerically distributed values like the dollar amount of sales transactions, you might determine that extremely large values are inaccurate. These extreme values might be replaced by a maximum value to ensure that aggregate calculations like average and standard deviation are not overly biased.

Conflicting information between fields (e.g., multiple addresses or significant deviation between a date-of-birth field and an age field) or between a field and applicable business logic (e.g., a transaction amount too large to be possible given constraints in the transaction process) are primary accuracy issues that should be addressed during the refined data stage. In some cases, generally when the percent of records with inaccurate values is small and unlikely to be significant, the appropriate remedy is to remove affected records. For many analyses, removing these records will not materially affect the results. In other cases, the best approach might be to reconcile conflicting information; for example, recalculating customer age using date-of-birth and the current date (or the dates of the events you want to analyze).

In many cases, resolving conflicting or inaccurate data fields in your refined data is best done using an explicit reference to time. Consider the multiple addresses problem in a customer dataset. Perhaps each address is (or was) correct, representing the various home addresses a person has had throughout her life. Assigning date ranges to the addresses might resolve the conflicts. Similarly, a transaction amount that violates current business logic might have occurred prior to that logic being enforced, in which case you might want to keep the transaction in the dataset to preserve the integrity of historical analyses.

The most useful notion of “time” can often require some care. For example, there might be a time when an action occurred, and a time when it was recognized. This is particularly common when dealing with bank transactions. In some cases, an abstract version number might be more appropriate than a timestamp. For example, for data generated by software, it might be more important to record the version of the software that was used rather than the time when the software ran. Similarly, in scientific experiments it might be more useful to know the version of a data file that was analyzed rather than the time that the analysis ran. In general, the appropriate choice of time or version often depends on the specifics of an analysis; therefore, it’s best to preserve (and document!) all the timestamps and version numbers that are available for a record.

Addressing scope issues

Stepping back from individual record field values, it is also important to design the scope of your refined datasets so that these datasets include the full required set of records and record fields. For example, suppose that your customer data is split into multiple datasets (one containing contact information, another containing transaction summaries, etc.) but most of your analyses involve all of these fields. It might make sense to create a fully blended dataset with all of these fields to streamline your analyses.

Perhaps the most important scope-related challenge is ensuring your refined datasets have understood population coverage. This means that a dataset should accurately express the relationship between the set of things represented by records in the dataset (people, objects, etc.) and the wider population of those things (e.g., all people and all objects).

Ideally, your dataset contains one and only one record representing each member of the population of things you want to analyze, but more likely, your dataset will contain a subset of records from the complete population. If the subset is a true, random subset of the wider population, your dataset can be analytically used to infer insights about the population as a whole. If, however, the subset of represented things in your dataset is not random—that is, it exhibits some kind of bias—you might be restricted on the kinds of analyses you can validly conduct. It’s beyond the scope of this book to discuss the statistical implications of bias between your dataset and the population of things it partially represents, but this is an important note to be aware of when designing refined datasets.