The general research finding of 10x differences in programmer productivity have led some people to want to measure the productivity of individual programmers in their organizations. For better or worse, the issues involved in measuring “live” programmers are quite different from the issues in measuring productivity for research purposes.
Software engineering studies normally measure productivity in terms of time to complete a specific task, or sometimes in terms of lines of code per effort-hour, staff-month, or some other measure of effort. When you try to measure productivity in any of these ways in a commercial setting, you’ll run into complications.
Software design is a nondeterministic activity, and different designers/developers will produce significantly different designs in response to a particular problem specification. Measuring productivity in lines of code per staff month (or equivalent) implicitly assumes that writing 10 times as much code to solve a particular problem means a programmer is 10 times more productive. That clearly is not always true. One programmer might have had a brilliant design insight that allowed him to satisfy the problem using only 10% as much code as a less capable programmer.
Some commentators have asserted that great programmers always write less code. Indeed, there probably is some correlation between programming excellence and code compactness, but I wouldn’t make that statement so broadly. I would say that great programmers strive to write clear code, and that usually translates to less code. Sometimes, however, the clearest, simplest, and most obvious design takes a little more code than a design that’s more “clever”—and in those cases I think the great programmer will write more code to avoid an overly clever design solution. One way or the other, the idea that productivity can be measured cleanly as “lines of code per staff month” is subject to problems.
The problem with measuring productivity in terms of lines of code per staff month is encapsulated by the old Dilbert joke about Wally coding himself a minivan. If you measure productivity in terms of volume of code generated, some people will optimize for that measure, i.e., they will find ways to write more lines of code, even if more lines of code aren’t needed. The problem here isn’t really with the specific “lines of code” measurement per se, but rather is an instance of the old management chestnut that “what gets measured gets done.” You need to be careful what you measure.
Some of the problems created by using “lines of code per staff month” can be avoided by measuring program size in function points. Function points are a “synthetic” measure of program size in which inputs, outputs, queries, and files are counted to determine program size. An inefficient design/coding style won’t generate more function points, so function points aren’t subject to the same issues as lines of code. They are, however, subject to more practical issues, namely that to get an accurate count of function points you need the services of a certified function point counter (which most organizations don’t have available), and the mapping between how function points are counted and individual work packages is rough enough that it becomes impractical to use them to ascertain the productivity of individual programmers.
Managers frequently say, “I always give my best programmer the most difficult/most complex sections of code to work on. His productivity on any measured basis might very well be low compared to programmers who get easier assignments, but my other programmers would take twice as long.” That’s also a legitimate phenomenon that does affect the attempt to define and measure productivity.
Difficulties like these have led many people to conclude that measuring individual productivity is so fraught with problems that no one should even try. I think it is possible to measure individual productivity meaningfully, as long as you keep several key factors in mind:
- Don’t expect any single dimensional measure of productivity to give you a very good picture of individual productivity.
Think about all the statistics that are collected in sports. We can’t even use a single measure to determine how good a hitter in baseball is. We consider batting average, home runs, runs batted in, on-base percentage, and other factors—and then we still argue about what the numbers mean. If we can’t measure a “good hitter” using a simple measure, why would we expect a simple measure to capture something as complex as individual programming productivity? What we need to do instead is use a combination of measures, which collectively will give us insights into individual productivities. (Measures could include on-time task completion percentage, manager evaluation on a scale of 1–10, peer evaluation on a scale of 1–10, lines of code per staff month, defects reported per line of code, defects fixed per line of code, bad fix injection rate, and so on.)
- Don’t expect any measures—whether single measures or a combination—to support fine-grain discriminations in productivity among individuals.
A good principle to keep in mind is that measures of individual productivity give you questions to ask but not the answers. Using measures of performance for, say, individual performance reviews is both bad management and bad statistics.
- Trends are usually more important than single-point measures.
Measures of individual productivity tend to be far less useful in comparing one individual to another than they are in seeing how one individual is progressing over time.
- Ask why you need to measure individual productivity at all.
In a research setting, researchers need to measure productivity to assess the relative effectiveness of different techniques, and their use of these measures is subject to far fewer problems than measuring individual productivity on real projects. In a real project environment, what do you want to use the measures for? Performance reviews? Not a good idea, for the reasons mentioned earlier. Task assignments? Most managers I talk with say they know who their star contributors are without measuring, and I believe them. Estimation? No, the variations caused by different design approaches, different task difficulty, and related factors make that an ineffective way to build up project estimates.
On real projects it’s hard to find a use for individual productivity measures that is both useful and statistically valid. In my experience, aside from research settings, the attempt to measure individual performance arises most often from a desire to do something with the measurements that isn’t statistically valid. So while I see the value of measuring individual performance in research settings, I think it’s difficult to find cases in which the measurement effort is justified on real projects.