Time for a TCP Benchmark Suite ?

This paper proposes that the networking community establish a TCP benchmark suite. We summarize the challenges in designing a benchmark suite, give a straw man proposal and list some findings with our benchmark. I. M OTIVATION Whenever there are design alternatives and no clear winners , Computer Scientists have always resorted to benchmarks. Th i is the case with the Computer Architecture community, where SPEC benchmarks have evolved over the years to reflect the diverse real world applications. This is the case with the Database community, where TPC (Transaction Processing Council) has been working on series of database system benchmarks since the 70s. This is also the case with systems such as file servers, disk drives, network switches, etc. Should we design a benchmark suite for TCP congestion control algorithms? One might argue that this is an absurd notion, because TCP is not a product. It is supposed to be a standard, with hopefully every detail enshrined in IETF RFC s, and implemented faithfully by everyone. On the other hand, the TCP protocol, algorithm and implementation never stayed static. Since TCP Reno, there hav been proposals such as NewReno, SACK, and FACK ([19]) that modify the loss recovery mechanisms in TCP. These proposals are already implemented in common operating systems such as Windows and Linux, but with configuration options to turn them off. There have also been proposals to change the AIMD congestion control algorithm, include TCP Vegas ([11]) and many proposals for high speed TCPs. Particularly , since it becomes apparent that TCP Reno fails to achieve high throughput on long distance fat pipe links, there have been many proposals for modifying the AIMD congestion control algorithm of TCP, with a few of them implemented in Linux already. Given the number of TCP variants, it is no surprise that the consumers of TCP, that is, application programmers and system administrators, are often confused. Some groups too k matter into their own hands and started their own benchmarking effort. For example, the Stanford Linear Accelerat o Center (SLAC) performed a series of tests comparing the high speed TCP variants [6]. However, the efforts of these groups are usually isolated and the conclusions are often not well understood. This paper proposes that the networking community establish a TCP benchmark suite. The benchmark consists of a set of network configurations (i.e. topologies, routing matrix and etc), a set of workloads (i.e. traffic generation rules), and set of metrics. The benchmark should have two modes: NS simulation mode, and hardware experiment mode. The two modes separate the artifact of implementation from the protocol i tself, and makes the test results repeatable. Ideally, the results of the various tests in the suite should be combined in certain fash ion to generate a single number, but we understand that it might be impossible to do so. There are three goals of the benchmark suite: • to crystallize the performance criteria that various interest groups attach to TCP. For example, networking researchers have traditionally listed aggregate throughp t and fairness as the performance criteria of TCP, whereas the application programmers typically use effective appli cation throughput and latency as the performance criteria. • to enhance understanding of the performance of various TCP variants under different scenarios, and give guidance to users about which variant to use depending on the network environments. • to help future TCP designers to verify the new algorithms in various environments. We recognize that establishing a good set of benchmark suite is a community effort. In this paper, we give a straw man proposal, and list some findings with our benchmark. We would like to use HotNets as a forum to discuss the various aspects of the benchmark. II. CHALLENGES IN DESIGNING A BENCHMARK There are at least three difficulties to design a benchmark for TCP congestion control algorithms. • How to define a set of quantitative, measurable and relevant performance metrics? • How to select representative scenarios from the space of topology and workload? • How to implement the benchmark testing in controlled and repeatable environments? We explain our approach to address these difficulties. A. Metrics for TCP congestion control We consider individual behavior of each flow, aggregate behavior of all the flows and the effects on the router. There are 11 metrics in our proposed benchmark (Many of them are in [12]). All the metrics here are quantitatively defined and can be calculated from the measured throughput, queuein g delay and loss. We categorize these metrics into three group s: equilibrium properties, stability properties and respons iveness properties. In the following subsections, we first explain the basic variables we measure. Then we explain how to calculate the metrics from these basic variables. Finally, we discuss the relevance of the metrics.