An Infrastructure for Worldwide Parallel Computing

With advances in high-speed networking, exploding interest in the World Wide Web (WWW), and widespread availability of Java, millions of Java-capable computers are now connected to the Internet. These heterogeneous laptops, personal computers, and workstations are emerging as a pool of distributed, platform-independent, Java virtual machines. The large-scale deployment of these systems provides the hardware needed for a national and international computing infrastructure, a virtual parallel computer that can be tapped for many uses. Although this infrastructure will have a great impact on distributed information processing and distributed computing, developing this middleware to support such wide area computing poses a host of thorny problems. The wide fluctuations in achievable network bandwidth, changing system availability, and significant differences in system capabilities make exploiting the large number of computing cycles available in Java-capable, WWW clients and servers extraordinarily difficult. One can effectively exploit these distributed computing cycles only when an infrastructure exists that can adapt to these variations, while providing reasonable performance and acceptable reliability. Simply put, the unreliable nature of wide area networks suggests the need for robust acquisition of distributed state information. In this thesis, we describe the robust information acquisition techniques, software design, prototype implementation, and performance evaluation of a Java-based infrastructure for supporting large-scale parallel computing on the Internet. We present results on the projected cost and performance of large scale infrastructures, and show that, given current technologies, it is feasible to construct hierarchies with 25,000 WWW browsers and 600 WWW servers. In addition, we validate our performance/reliability models via experiments, show that the time scale of world view maintenance and task scheduling are on the order of minutes, and that prediction techniques can enable the maintenance of a distributed hierarchy for large scale application scheduling.