The process of cleaning data involves tidying the data, which usually results in making the dataset smaller because we have cleaned out some of the dirty data. What makes data dirty?

Dirty data can be due to invalid data, which is data that is false, incomplete, or doesn't conform to the accepted standard. An example of invalid data could be formatting errors, or data that is out of an acceptable range. Invalid data could also have the wrong type. For example, the Asterix is invalid because the acceptable formatted data is for letters only, so it can be removed.

Dirty data can be due to missing data, which is data where no value is stored. An example of missing data is data that has not been stored due to a faulty sensor. We can see that some data is missing, so it is removed from consideration.

Dirty data could also have null values. If data has null values, then programs may respond differently to the data on that basis. The nulls will need to be considered in order to remove their impact, or they could be removed.

Dirty data could be due to duplicated data. This could occur due to invalid collection of data by a sensor, for example. Duplicated data needs to be fixed because it can increase the amount of total storage space taken up by the data.

Altogether, data can be very messy if it has any of these characteristics, and in some unfortunate cases it can have some, or even all of these characteristics. In machine learning, we need to remove the influence of dirty data because it can produce bad results.

Data quality is one of the issues that organizations can find very difficult to resolve. For some people, the reasons are technical in nature; the data can be very difficult to clean because of the technology itself. One of the most pernicious causes of poor data quality is due to process and user error.

What are the consequences of poor data quality? Simply put, it's bad for a business to have inaccurate or incomplete data. Poor data quality is ubiquitous in organizations, partly due to a breakdown in communication between CIOs and business managers.

In the following sections, we will focus on an R package known as dplyr.