Eye tracking and program comprehension

Reading and comprehending code is different from reading regular text. And eye tracking, which measures where we look, for how long, and how much mental effort we exert, can provide crucial data. The talk will survey some achievements and suggest future directions. Code is hardly ever written in a vacuum. New code must dovetail with existing code. Much code is taken from libraries and frameworks. Developers therefore spend more time reading and trying to understand existing code than writing new code. Code comprehension is also required for fixing bugs. As a consequence, the time it takes to comprehend code is a major driver of development costs. Code comprehension is often thought of as the reciprocal of code complexity: the more complex the code, the harder it is to comprehend. But experience with code comprehension studies indicates that the difficulty of comprehension actually reflects the confluence of three distinct factors: the code, the reader, and the moment. The code's complexity is indeed a factor, but not the only one and perhaps not even the major one. The experience and domain knowledge of the person reading the code is also very important; code that is incomprehensible to one may be crystal clear to another. And various transient effects --- such as fatigue, priming, or being preoccupied --- also take their toll. Studies of code comprehension try to untangle all these effects. One of the tools sometimes used is eye tracking. An early and often-repeated result is that reading code is very different from reading regular text. For example, it has been observed that in many cases reading code includes episodes of scanning the code in a linear fashion. There may be multiple such episodes, or none. In addition there are instances of jumping back to look at the beginning of the code, or jumping to the end. It has been conjectured that these reading patterns represent getting an overview of the code, rechecking a function signature or the declaration of variables, and a preview of a function's return value. Some studies have used eye tracking to characterize the code comprehension activities of different people. A relatively popular topic has been to compare code reading strategies employed by experts and novices. Several studies have found that experts distribute their attention differently and perhaps more efficiently. As another example, Sharafi et al. found that while men and women achieve similar performance in identifying variable names, their strategies might differ. It appeared that women spend more time on checking alternatives, whereas men somehow focus more directly on the correct answers. Working with Ahmad Jbara, we have constructed the first quantitative model of comprehension effort that is based on eye tracking data. We conjectured that regular code, comprising repetitions of the same basic structure, would be easier to comprehend than equivalent non-regular code. The idea was that once the first instance of the recurring structure is understood, this understanding can be leveraged for understanding the subsequent instances. Using eye tracking we showed that this is true, and indeed less and less time is spent on repeated instances. An important aspect of some of these examples is that they represent basic science. We learn something interesting about how our world (the world of developers and code) works, even if it does not have immediate implications that practitioners can utilize tomorrow. Software development is a complex process, influenced by myriad factors. There is a lot we don't know yet. So a focus on basic science is appropriate, and should be encouraged. Such scientific knowledge can then serve as the scientific basis for software engineering in ways that were not anticipated in advance. Part of this is to study not only the software engineering itself, but also the methodology we use to study it. One example that has received some attention is the metrics that are collected in eye tracking data. Should it be the number of fixations, cumulative fixation time, saccades, the convex hull of all fixations, attributes of the scan path (such as regressions), or some other metrics? Interestingly only a few studies have used pupil diameter to estimate mental effort, so this is a promising approach for new results. Finally, further progress will be accelerated if we build and share tools. Research tools, such as software for analyzing eye tracking data, capture and propagate the experience of whoever created them. And using the same tools improves reproducibility.