We compared the results from the developer and reporter surveys to find out whether they agree on what is important in bug reports.
First, we compared the information developers use to resolve bugs (question D1) and the information reporters provide (R1). In Figure 24-3, items in the left column are sorted decreasingly by the percentage of developers who have used them, and items in the right column are sorted decreasingly by the percentage of reporters who have provided them. Lines connect the same items across columns and indicate the agreement (or disagreement) between developers and reporters on that particular item. Figure 24-3 shows that the results match only for the top three items and the last one. In between there are many disagreements, and the most notable ones are for stack traces, test cases, code examples, product, and operating system. Overall, the Spearman correlation[33] between what developers use and what reporters provide was 0.321, far from ideal.
Next, we checked whether reporters provide the information that is most important for developers. In Figure 24-4, the left column corresponds to the importance of an item for developers (measured by questions D2 and D1), and the right column to the percentage of reporters who provided an item (R1). Developers and reporters still agree on the first and last item, but overall the disagreement increased. The Spearman correlation of ‒0.035 between what developers consider as important and what reporters provide shows a huge gap. In particular, it indicates that reporters do not focus on the information important for developers.
Interestingly, Figure 24-5 shows that most reporters know which information developers need. In other words, ignorance among reporters cannot be blamed for the aforementioned information mismatch. As before, the left column corresponds to the importance of items for developers; the right column now shows what reporters expect to be most relevant (question R3). Overall there is a strong agreement; the only notable disagreement is for screenshots. This is confirmed by the Spearman correlation of 0.839, indicating a very strong relation between what developers and reporters consider as important.
To sum up, to improve bug reporting systems, one could tell users while they are reporting a bug what information is important (e.g., screenshots). At the same time, systems should provide better tools to collect important information, because often this information is difficult to obtain for users.
[33] Spearman correlation is a measure of strength of the association between two variables. Its value ranges from ‒1 to +1. Values closer to ‒1 or +1 indicate a strong relationship, while 0 suggests there is no relationship between the variables. The sign indicates whether the association is in the same (+) or opposite (‒) directions.