Determining Data Needs

It is important to understand the basics of data analysis and why an organization would invest in the process of analyzing data. The data analysis process involves collecting data by using an appropriate tool or an application that permits exploration of the data as well as possibly identifying trends or patterns in the data. With the help of such information, conscious decisions can be made.

Data analysis involves the following phases or steps:

1. Requirements gathering for data: It is a good idea to consider the need to perform data analysis and the outcomes that may be achieved by analyzing the data collected. That is, it is a good idea to identify the reason for performing data analysis. During the requirements gathering phase, also known as the preparation phase, you need to decide what to examine, the data structure, and the volume of data. In addition, it is important to understand what/why you are examining and what evaluation methodologies you expect to leverage to perform your analysis.

2. Collection of data: Once you have an understanding of what you need to evaluate and what the outcomes/results will be, the next phase is data collection. As data is collected, it needs to be organized or processed for analysis. Because data might be collected from multiple sources, it is a good practice to keep a log of data sources and collection dates/times.

3. Cleansing of data: Any kind of data gathered for analysis purposes has to be cleaned so there is minimal noise, redundancy, and duplication in the data and so pertinent information can be extracted as a result of analysis. Data gathered may include white spaces, duplicate records, and errors. Data should ideally be error free and cleansed. The cleansing phase is completed prior to the analysis phase because, after cleansing the data, the results of analysis will be closer to the expected results.

4. Analysis of data: After data is gathered, processed, and cleansed, it is ready for analysis. As you analyze/manipulate data, you can identify whether you have all the data required for successful analysis and outcomes or whether you need to gather more data for effective decision making. At this stage, the tools and software for data analysis help you interpret, understand, and make conclusions based on well-defined needs.

5. Interpretation of data: After data is analyzed, results have to be interpreted. Data interpretation involves assigning meaning to data such that it can be leveraged for driving/discussing outcomes. Interpretation can be done leveraging the qualitative or quantitative methods.

6. Visualization of data: It is common to leverage visualizations to show the analysis and conclusions in the form of graphs and charts. Illustrating data graphically helps make it easy to understand and process. Visualizations may be used for discovering unknown trends and facts. By comparing datasets and relationships, you can identify a way to discover meaningful outcomes. The most common way to visualize data is via reports and dashboards. Reports can be in simple English or may include charts or tables for brevity.

Review/Refine Business Questions

In order to transform data into business insights and drive decisions, it is important to prepare and follow a set of questions that provides a direction for the analysis performed. These questions will form the basis from which you would acquire answers prior to starting the process of data gathering. These questions can be based on an organization’s goals, strategy, budget, addressable market, and target clientele. Some of the questions to be reviewed are as follows:

• What is the problem we are trying to solve or outcome we are trying to achieve?

• Who is the sponsor for this activity?

• What are the resources (people, time, budget) that we should think about?

• What are the different sources from which pertinent data can be collected?

• How can we assure quality of data?

• Which techniques can possibly be applied for data analysis?

• What end users or stakeholders might benefit from the analysis?

• What type of visualizations might help articulate the findings effectively?

• What type of software would help with collection, analysis, and visualization?

• Are there other things we should consider to ensure that the outcomes are in line with the business goals?

• What are the standard/key performance indicators that may define success criteria?

Data Collection Sources

At the heart of data analysis is data collection. Data may be well organized (for example, in SQL tables) or unorganized (for example, in figures or files), and it may consist of facts.

Broadly, there are two types of data: statistical and nonstatistical data. Statistical sources include official surveys and the census. Nonstatistical sources include administrative records, particularly in the private sector. Additional data sources are categorized on the basis of the methods for collection:

• Primary or secondary

• Internal or external sources

Figure 11.1 illustrates sources for collection of data, which include the following:

• Primary data sources: Data collected firsthand from a researcher or surveyor is known as primary data. This kind of data is gathered for the first time and may be from internal or external sources.

• Secondary data sources: Data that was previously gathered, processed, published, or analyzed is known as secondary data. This data may be gathered from internal or external sources by investigators, researchers, or surveyors for carrying out statistical analysis and may be referred to as secondhand data.

• Internal data sources: Internal data sources, as the name suggests, are internal to an organization (for example, archives, records, and other sources within the organization). The organization (in most cases) has direct control over these data sources.

• External data sources: External data sources are outside the organization, sometimes also referred to as public sources (for example, census data). The organization may not have direct control over or access to them.

Gap Analysis

A gap analysis is typically performed to compare the current state to a future state, and an action plan is created to get from the current state to the desired state. Organizations perform gap analysis to enhance their competence and competitive advantage in terms of their products, services, or processes. Gap analysis enables business leaders to make conscious decisions to optimize the ways in which money, human, and time resources are spent.

Gap analysis yields insights into the following aspects:

• Current/present state

• Desired/future state

• What needs to be done (action plan) to bridge the gap

These aspects can be leveraged to drive performance improvements. Figure 11.2 illustrates the basics of gap analysis.

As Figure 11.2 illustrates, the gap separates the present or current condition/state from the desired or future condition/state. In this example, an organization wishes to bridge the gap between the current state of $1 million in sales year on year (YoY) and the desired state of $6 million in sales YoY by the end of the next 2 years.

Gap analysis can assist in identifying specific areas for enhancement, whether related to optimization of the workforce, efficiency, strategy, or performance.

The following steps are involved in gap analysis:

1. Understand or determine the current or present state.

2. Determine the future or desired state.

3. Understand the gap between the current/present state and the future/desired state.

4. Implement a plan of action or required steps (also known as remedies) to reduce or eliminate the gap and improve performance, productivity, efficiency, and so on.

Several types of gaps exist, and how they are analyzed differs:

• Strategy or performance gap: This type of gap is analyzed in terms of actual performance vs. expected performance.

• Market or product gap: This type of gap is analyzed in terms of actual sales vs. budgeted sales for a product or service. It may also be analyzed from a competitive perspective to improve a product/service to compete and increase its market share.

• Manpower gap: This type of gap is analyzed based on the actual number of work hours vs. a projected number of work hours.

• Profit gap: This type of gap is analyzed in terms of actual profit vs. target profit.

The gap analysis involves the following steps:

1. Determine what data to analyze.

2. Identify the methods to use to assess the present state.

3. Identify the future state and create an action plan to move from the current state to the future state.

4. Record the attributes that may influence the success or failure from a gap perspective.

Let’s consider an example of improving sales figures for a product that your organization wishes to release a new version of. In this scenario, you would need to use gap analysis and develop a plan of action (or action plan).

The current state is the number of products being sold, and the data sources that can be taken into consideration to carry out gap analysis are the historical and current sales figures as well as customer feedback. This information acts as a baseline for the future state or the number of products your organization would like to sell. Based on current sales numbers, you can get an idea of the gap between what is currently being done (for example, sales training, product enhancements, market demand) and create a plan to attain the future-state sales numbers. You should document this plan and ensure that all attributes that influence sales (such as training, market demand, and product features) are communicated effectively within the organization.

Trend Analysis

Trend analysis is the process of gathering information from diverse periods (via time series data analysis) to drive insights about trends or patterns. Trend analysis is based on comparisons of data over specific time periods in order to spot patterns or trends.

In a business context, trend analysis can be leveraged for a number of purposes:

• Facilitating comparison: Based on market trend data, analysts can compare two or more organizations in terms of their sales performance.

• Investment analysis: Analysts can conduct a comparative study of the financial performance of an organization over a period of time.

• Cost and revenue analysis: Analysts can measure the profitability of an organization over a period of time.

Figure 11.3 gives an overview of trend analysis in which the analyst studies trends of sales of a product over different periods to determine future demand.

Trend analysis is used across industries, typically to ensure that an organization stays ahead of competition by understanding consumer buying and spending trends as well as trends related to products and features.

Performance Analysis

Performance analysis involves studying or comparing the performance of a particular activity or process in order to identify strengths and weaknesses. Performance analysis from a human resources point of view can assist in reviewing the contribution of employees toward a given assignment. It could be used to analyze business performance or sports performance, for example. In either case, an individual’s performance is analyzed, and their strengths and weaknesses are noted to improve performance (in terms of sales or race times) in the near or far future.

Performance analysis can be used to improve profitability, increase employee engagement by ensuring that employees’ efforts are valued and rewarded, and set a baseline to drive performance measurements across organizations.

Exploratory Data Analysis

Data scientists and statisticians use exploratory data analysis for analyzing and investigating sets of data and summarizing their major characteristics. In the process, they leverage various methods of data visualization. In a nutshell, this type of data analysis allows data scientists and statisticians to uncover patterns as well as insights, often with the aid of visual methods. It also assists in testing a hypothesis, spotting anomalies, and checking assumptions. Some of the key aspects of exploratory analysis can be categorized based on the outcomes expected, such as:

• Hypothesis: What patterns are emerging, and how can I explain them?

• Graphs and charts: What can I see here?

• Grouping (or ungrouping) data: How can the data be sliced and diced as well as clustered?

• Model building: What patterns/trends/shapes can be observed, and why?

Use of Descriptive Statistics to Determine Observations

Descriptive statistics involves describing the characteristics of a dataset or sample data and often leverages graphical methods to illustrate data characteristics. Descriptive statistics is usually used for the following purposes:

• To offer characteristics of the variables in a dataset

• To highlight the potential relationships between two or more variables

The most common measures of descriptive statistics are central tendency, dispersion, and frequency, which are discussed in Chapter 10. Additional methods are shown in Figure 11.8.

Descriptive analysis methods can be described or depicted using:

• Measures of position: A measure of position makes it possible to determine the position of a particular data value within a given dataset. Common measures include standard scores, quartiles, and percentiles.

• Contingency tables: These tables describe relationships between two or more categorical values in a tabular matrix arrangement.

• Scatter plots: Also known as scatter charts, scatter plots illustrate relationships between two values across X- and Y-axis plots.

• Histograms: A histogram is used to graphically illustrate the distribution of numeric values as a series of bars.

• Sociograms: A sociogram is a graphic representation for finding relationships within a group of people.

Advantages of adopting descriptive statistics are as follows:

• It provides a high level of neutrality and objectivity.

• It provides a wide representation of an event leveraging the variable characteristics.

Link Analysis

Link analysis is a data mining mechanism that can help structure data as a network of unified nodes and links for identifying and analyzing connections and relationships. Link analysis allows analysts to identify connections and association patterns within the nodes and links of a network. Examples of link analysis are semantic networks and social networks.

Link analysis considers the following key aspects:

• Network: This is a unified body consisting of interconnected nodes and links.

• Node: This is a point that represents a real-world object, such as a person or place.

• Link: This shows the relationship or connection between nodes.

For example, a crime investigator would use clues and evidence to set up a network of suspects and link them to the victim via their relationships, as shown in Figure 11.9. As you can see, for each object/node, there is additional information.

Link analysis can be useful for:

• Analytical applications that leverage network graphs for drawing conclusions

• Assessing the influence of individuals in a social network

• Process or path optimization (such as an airline carrier optimizing flight routes for better fuel efficiency)