Benford's law is a popular empirical law that states that the first digits of a population of data will follow a specific logarithmic distribution.
This law was observed by Frank Benford around 1938 and since then has gained increasing popularity as a way to detect anomalous alterations in a population of data.
Basically, testing a population against Benford's law means verifying that the given population respects this law. If deviations are discovered, the law performs further analysis for items related to those deviations.
In this recipe, we will test a population of e-commerce orders against the law, focusing on items deviating from the expected distribution.
This recipe will use functions from the well-documented benford.analysis package by Carlos Cinelli.
We therefore need to install and load this package:
install.packages("benford.analysis") library(benford.analysis)
In our example, we will use a data frame that stores e-commerce orders, provided within the book as an .Rdata file.
In order to make it available within your environment, we need to load this file by running the following command (assuming the file is in your current working directory):
load("ecommerce_orders_list.Rdata")
- Perform the Benford test on the order amounts:
benford_test <- benford(ecommerce_orders_list$order_amount,1) - Plot the test analysis:
plot(benford_test)This will result in the following plot:
- Highlight digits deviating from expected distribution:
suspectsTable(benford_test)This will produce a table showing, for each digit, the absolute differences between expected and observed frequencies. The first digits will therefore be the anomalous ones:
> suspectsTable(benford_test) digits absolute.diff 1: 5 4860.8974 2: 9 3764.0664 3: 1 2876.4653 4: 2 2870.4985 5: 3 2856.0362 6: 4 2706.3959 7: 7 1567.3235 8: 6 1300.7127 9: 8 200.4623
- Define a function to extrapolate the first digit from each amount:
left = function (string,char) { substr(string,1,char)}
- Extrapolate the first digit from each amount:
ecommerce_orders_list$first_digit <- left(ecommerce_orders_list$order_amount,1) - Filter amounts starting with the suspected digit:
suspects_orders <- subset(ecommerce_orders_list,first_digit == 5)
In step 1, we perform the Benford test on the order amounts. In this step, we apply the benford() function to the amounts.
Applying this function means evaluating the distribution of the first digits of amounts against the expected Benford distribution.
The function will result in the production of the following objects:
In step 2, we plot the test results. Running plot on the object resulting from the benford() function will result in a plot showing the following (from upper-left corner to bottom-right corner):
- First digit distribution
- Results of the second-order test
- Summation distribution for each digit
- Results of the chi-squared test
- Summation differences
If you look carefully at these plots, you will understand which digits show a distribution significantly different from the one expected by the Benford law. In order to have a sounder base for our consideration, we need to look at the suspects table, showing absolute differences between expected and observed frequencies. This is what we will do in the next step.
In step 3, we highlight suspects digits. Using suspectsTable() we can easily discover which digits present the greater deviation from the expected distribution.
Looking at the suspects table, we can see that number 5 shows up as the first variable within our table. In the next step, we will focus our attention on the orders with amounts that have this digit as the first digit.
In step 4, we define a function to extrapolate the first digit from each amount. This function leverages the substr() function from the stringr() package and extracts the first digit from the number passed to it as an argument.
In step 5, we add a new column to the investigated dataset, where the first digit is extrapolated.
In step 6, we filter amounts starting with the suspected digit.
After applying the left function to our sequence of amounts, we can now filter the dataset, retaining only rows whose amounts have 5 as the first digit. We will now be able to perform analytical testing procedures on those items.