A methodology for making early comparative architecture performance evaluations

Complex and expensive systems' development suffers from a lack of method for making good system-architecture-selection decisions early in the development process. Failure to make a good system-architecture-selection decision increases the risk that a development effort will not meet cost, performance and schedule goals. This research provides a method to mitigate that risk based on the idea that a development can be characterized as the management of uncertainties in a probabilistic experiment. The method developed shows how to estimate the probability that an arbitrary implementation of one system-architecture will perform better than an arbitrary implementation of an alternate system architecture. The analysis technique presented acknowledges that many implementation uncertainties exist at system-architecture-selection time and identifies steps that can be used to characterize these uncertainties. The process by which uncertainty descriptions are combined into architectural performance descriptions is presented. Once all alternative system architecture performance descriptions are developed relative system architecture performance comparisons can be made. After the analysis technique is described, three examples are considered. The first example is a simple three tier web-enabled database application. This small web application is used to illustrate the analysis method and demonstrate some methods for characterizing uncertainties. The next two examples are more complex. These examples expose a broader set of uncertainties and show how to handle cases where large numbers of uncertainties exist. Sections on validation of results follow. The dissertation concludes with a list of future research opportunities in this area.