Performance evaluation of complex multi-thread applications through multi-level critical path analysis

The sporadic latency problems of multi-thread applications remain a major concern for developers. A performance issue may be caused by internal factors like an error in the code or an inadequate database design, or external factors such as threads imbalance, resource contention or system overload. In complex applications such as the Chromium browser, which is our focus in this paper, being able to collect precise unified information from several layers of the application/system is the first challenge. There exists a built-in tracer in Chromium, however it provides a limited view of the application execution. The correlated analysis of the data collected from the multiple execution levels of the application is another challenge. In this paper, after explaining our tracing method, for collecting data from the application and the operating system kernel simultaneously, we translate the tracing data, collected from several layers of the system, into a unified model. We then use this model to perform kernel-level control flow analysis for user-level browser actions, to detect latency issues and identify their main causes. We show the effectiveness of the proposed method by detecting three Chromium performance bugs and finding their root-causes. Although our use cases are based on the Chromium browser, the proposed method and lessons learned can be useful when it comes to analysis of many other complex multi-thread applications.