The results of our survey are summarized in Figure 24-1 (for developers) and Figure 24-2 (for reporters).
In the figures, responses for each item are annotated as bars (
) , which can be interpreted as follows (explained with D1 and D2 as
examples):
The larger the black bar is in proportion to the gray bar, the higher is the
corresponding item’s importance in the developers’ perspective. The importance of every
item is also listed in parentheses. For example, Figure 24-1 shows
that developers consider steps to reproduce (
, 83%) to be more important than build information
(
, 8%).
Figure 24-1 shows that the most widely used items across projects are steps to reproduce the bug, observed and expected behavior, stack traces, and test cases. Information rarely used by developers include hardware and severity. Eclipse and Mozilla developers liked screenshots, whereas Apache and Eclipse developers more often used code examples and stack traces.
For the importance of items, steps to reproduce the bug stand out clearly as most important. Next in line are stack traces and test cases, both of which help to narrow down the search space for defects. Observed behavior mimics, albeit weakly, steps to reproduce the bug, which is why it may be rated as important. Screenshots were rated as high, but often are helpful only for a subset of bugs, e.g., GUI errors.
Smaller surprises in the results include the relative low importance of items such as expected behavior, code examples, summary, and mandatory fields such as version, operating system, product, and hardware. As pointed out by a Mozilla developer, not all projects need the information that is provided by mandatory fields:
“That’s why product and usually even component information is irrelevant to me and that hardware and to some degree [OS] fields are rarely needed as most our bugs are usually found in all platforms.”
In any case, we advise caution when interpreting these results. Items with low importance in our survey are not totally irrelevant, because they still might be needed to understand, reproduce, or triage bugs.
The items provided by most reporters are listed in the first part of Figure 24-2. As expected, observed and expected behavior and steps to reproduce the bug rank highest. Only a few users added stack traces, code examples, and test cases to their bug reports. An explanation might be the difficulty of providing these items, which we measured by the percentage of people who selected an item as difficult to provide in bug reports; the difficulty is listed in parentheses. All three items rank among the more difficult items, with test cases being the most difficult item. Surprisingly, steps to reproduce the bug are considered difficult as well, as is the component. For the latter, reporters revealed in their comments that often it is impossible for them to locate the component in which a bug occurs.
Among the items considered to be most helpful to developers, reporters ranked steps to reproduce the bug and test cases highest. Comparing the results for test cases among all three questions reveals that most reporters consider them to be helpful, but only a few provide them because they are most difficult to provide. This suggests that capture/replay tools that record test cases should be integrated into bug tracking systems. A similar but weaker observation can be made for stack traces, which are often hidden in logfiles and difficult to find. On the other hand, both developers and reporters consider the component to be only marginally important; furthermore, as already mentioned, it is rather difficult to provide.