Case Studies for Complexity Pattern Identification

The INCOSE Complex Systems Working Group Heuristics Team has selected 67 Principles and Heuristics that are considered to be particularly relevant to Complex Systems. These have been incorporated into a Difficulty Assessment Tool that prioritizes the list of Principles and Heuristics based on scoring of a matrix of four Difficulty Elements and six System Elements (to characterize the nature of the complexity). The purpose of this paper is to describe an initial assessment of the effectiveness and usefulness of the Difficulty Assessment Tool. The Tool has been used to assess eight Case Studies by five assessment teams — one with three people working together, one with two people and the remaining three with individual assessments. The results of these assessments have been compared using four different correlation methods, using the total weighted Heuristic score, the maximum weighted Heuristic score, a Match / Mismatch analysis of the top fifteen and bottom seven Heuristics, and a difference ranking between pairs of assessors of all 67 Principles and Heuristics. The last two assessment methods are shown to be more insightful. The assessment teams then reviewed the relevance of the highest and lowest-ranked Principles and Heuristics to the full Case Study definitions (Problem and Outcome). There is good agreement of relevance for the highest-ranking Principles and Heuristics, less so for the lowest-ranking ones. Based on this initial assessment, the DAT shows promise to help people develop complex systems. The paper concludes with recommendations for further assessment of the Difficulty Assessment Tool.