Model verification and validation (V&V) is an enabling methodology for the development of numerical models that can be used to make predictions with quantified accuracy and confidence. Model V&V procedures are needed to reduce the time, cost and risk associated with component and full-scale testing of engineered systems. Quantifying the confidence and predictive accuracy of model calculations provides the decision-maker with the information necessary for making high-consequence decisions. One of the key steps in a model validation project is to compare model predictions and experimental data given uncertainties in both. This step is critical because it defines how the model will be challenged against the physical data relative to the intended use of the model. This paper briefly reviews the main concepts involved in model V&V, introduces a simple measure termed the z-metric, and compares the z-metric with another measure called the area metric.

The objective of this investigation was to develop probabilistic finite element (FE) models of the anterior longitudinal ligament (ALL) and posterior longitudinal ligament (PLL) of the cervical spine that incorporate the natural variability of biological specimens. In addition to the model development, a rigorous validation methodology was developed to quantify model performance. Experimental data for the geometry and dynamic properties of the ALL and PLL were used to create probabilistic FE models capable of predicting not only the mean dynamic relaxation response but also the observed experimental variation of that response. The probabilistic FE model uses a quasilinear viscoelastic material constitutive model to capture the time-dependent behaviour of the ligaments. The probabilistic analysis approach yields a statistical distribution for the model-predicted response at each time point rather than a single deterministic quantity (e.g. ligament force) and that response can be statistically compared to experimental data for validation. A quantitative metric that compares the cumulative distribution functions of the experimental data and model response is computed for both the ALL and PLL throughout the time histories and is used to quantify model performance.

This presentation provides an overview of the AIAA Special Session 22-NDA-4 entitled “Verification and Validation Methods.” Specifically, the talk introduces a series of papers addressing the role of uncertainty quantification in validation assessment. All but one of the papers represents current activities within an ASME technical committee that is developing guidelines for performing verification and validation for computational solid mechanics models.

NASA/NESSUS 6.2c is a general-purpose, probabilistic analysis program that computes probability of failure and probabilistic sensitivity measures of engineered systems. Because NASA/NESSUS uses highly computationally efficient and accurate analysis techniques, probabilistic solutions can be obtained even for extremely large and complex models. Once the probabilistic response is quantified, the results can be used to support risk-informed decisions regarding reliability for safety-critical and one-of-a-kind systems, as well as for maintaining a level of quality while reducing manufacturing costs for larger-quantity products. NASA/NESSUS has been successfully applied to a diverse range of problems in aerospace, gas turbine engines, biomechanics, pipelines, defense, weaponry, and infrastructure. This program combines state-of-the-art probabilistic algorithms with general-purpose structural analysis and lifting methods to compute the probabilistic response and reliability of engineered structures. Uncertainties in load, material properties, geometry, boundary conditions, and initial conditions can be simulated. The structural analysis methods include non-linear finite-element methods, heat-transfer analysis, polymer/ceramic matrix composite analysis, monolithic (conventional metallic) materials life-prediction methodologies, boundary element methods, and user-written subroutines. Several probabilistic algorithms are available such as the advanced mean value method and the adaptive importance sampling method. NASA/NESSUS 6.2c is structured in a modular format with 15 elements.

Cervical spine injuries can occur as a result of large inertial forces such as those experienced by military aviators during ejection or evasive maneuvers. Mitigating the risk of injury during these types of extreme scenarios requires a fundamental understanding of the injury mechanisms associated with the transferral of loads through the human body and the ensuing kinematic response to these loads. This understanding can then be used to develop operational procedures, flight deck equipment and personnel protection gear to minimize the risk of injury. However, the wide range of loading scenarios, anatomical differences associated with aviator size and gender, and the obvious inability to perform full scale testing necessitate the use of numerical modelling and simulation to predict the uncertain behaviour and response of the human body to extreme loads. The credibility in these predictions is established through the application of formal model verification and validation (V&V) practices and procedures. A critical aspect to V&V applied to highly complex models where uncertainties are large and testing is especially difficult is the use of a validation hierarchy where the full system model is subdivided into its constituent subassemblies and components. Model V&V is then applied to each submodel in the hierarchy such that the error and confidence in the full system prediction can be quantified. This paper will briefly describe a model V&V process currently under development by the American Society of Mechanical Engineers and then provide an example of the hierarchical process applied to the development of a cervical spine model for predicting the risk of injury to military aviators. 1.0 INTRODUCTION Numerical models are now routinely used to simulate complex behaviour in solid mechanics, dynamics, hydrodynamics, heat conduction, fluid flow, transport, chemistry, biology, and acoustics. These models can produce response measures such as deformation, stress, and velocity histories, and failure measures such as fatigue, fracture, and wear. The high-fidelity of these models, however, should not be confused with credibility, i.e., there is generally not a one to one relationship between the two. Model fidelity is the result of modelling tools, simulation software and computational speed; credibility, on the other hand, is the result of model verification and validation (V&V). Model V&V are the primary processes for quantifying and building credibility in numerical models. Verification is the process of determining that a model implementation accurately represents the developer’s Thacker, B.H.; Francis, W.L.; Nicolella, D.P. (2007) Model Validation and Uncertainty Quantification Applied to Cervical Spine Injury Assessment. In Computational Uncertainty in Military Vehicle Design (pp. 26-1 – 26-30). Meeting Proceedings RTO-MP-AVT-147, Paper 26. Neuilly-sur-Seine, France: RTO. Available from: http://www.rto.nato.int. UNCLASSIFIED/UNLIMITED UNCLASSIFIED/UNLIMITED Model Validation and Uncertainty Quantification Applied to Cervical Spine Injury Assessment 26 2 RTO-MP-AVT-147 conceptual description of the model and its solution. Validation is the process of determining the degree to which a model is an accurate representation of the real world from the perspective of the intended uses of the model [1]. Model V&V are processes that accumulate evidence of a model’s correctness or accuracy for a specific scenario; thus, V&V cannot prove that a model is correct and accurate for all possible scenarios, but rather provide evidence that the model is sufficiently accurate for its intended use. Therefore, model V&V is a process that is never fully completed, but rather concluded when acceptable accuracy is achieved. Most complex systems will comprise multiple (and complex) subsystems and components, each of which must be modelled and validated. It is not entirely uncommon for model developers to attempt to validate full system models directly from any available test data. This approach can be problematic if there are a large number of components or if subsystem models that contain complex connections or interfaces, energy dissipation mechanisms, and highly nonlinear behaviour. If there is poor agreement between the prediction and the experiment, it can be difficult, if not impossible, to isolate which subsystem is responsible for the discrepancy. Even if good agreement between prediction and experiment is observed, it is still possible that the model quality could be poor because of error cancellation among the subsystem models. Therefore, a hierarchical strategy must be used to conduct a careful sequence of submodel validations to build confidence in the ability of the full system model to provide accurate predictions. Uncertainty and error quantification play a key role in model V&V. Nondeterminism refers to the existence of errors and uncertainties in the outputs of computational simulations due to uncertainties in the model. Likewise, the measurements that are made to validate these simulation outputs also contain errors and uncertainties. While the experimental data are used as the reference for comparison, the V&V process does not presume the experiment to be “right” or even more accurate than the simulation. Instead, the goal is to quantify the uncertainties in both experimental and simulation results such that the predictive accuracy of the model can be quantified. Cervical spine injuries can occur as a result of impact or from large inertial forces such as those experienced by military pilots during ejections, carrier landings, and ditchings. Understanding the mechanisms and risk of cervical spine injury to military pilots during high-risk situations motivated the development of a numerical modelling capability that could be used to quantify the risk of injury and explore design modifications to reduce this risk as much as possible. In addition, the effect of gender on the risk of injury was also of interest due to the increasing number of female aviators. In biological systems there is a great deal of uncertainty associated with the environment in which the structure is required to function as well as physical and mechanical properties and geometry of bone, ligaments, cartilage, joint and muscle. These uncertainties have a direct effect on the behavior and response of the system. The consequences associated with cervical spine injuries are also usually severe. Therefore, it is of great interest to have a validated predictive tool that can be used to design occupant safety systems to minimize the probability of injury. To do this, the designer must have quantified knowledge of the probability of injury due to different loading scenarios, and also understand which model parameters contribute the most to the injury probability. 2.0 MODEL VERIFICATION AND VALIDATION Model verification and validation is undertaken to quantify confidence and build credibility in a numerical model for the purpose of making a prediction. Ref. [1] defines prediction as “use of a computational model to foretell the state of a physical system under conditions for which the computational model has not been UNCLASSIFIED/UNLIMITED UNCLASSIFIED/UNLIMITED Model Validation and Uncertainty Quantification Applied to Cervical Spine Injury Assessment RTO-MP-AVT-147 26 3 validated.” The predictive accuracy of the model must therefore reflect the strength of the inference being made from the validation database to the prediction. If needed, the predictive accuracy of the model can be improved as supported by additional experiments, information, or experience. Verification is concerned with identifying and removing errors in the model by comparing numerical solutions to analytical or highly accurate benchmark solutions. Validation, on the other hand, is concerned with quantifying the accuracy of the model by comparing numerical solutions to experimental data. In short, verification deals with the mathematics associated with the model, whereas validation deals with the physics associated with the model. Obviously, verification must be performed before validation. It is important to define and differentiate between the terms “code” and “model” in the context used herein. A code is the computer implementation of algorithms in specific computer software developed to facilitate the formulation and approximate solution of a class of models. A model includes the conceptual, mathematical, and numerical representation of a specific physical scenario and the equations comprising the model are solved using the algorithms in the code. 2.1 Validation Hierarchy Figure 1 shows a schematic of generic model decomposition into a hierarchy of submodels. The top tier represents the complete system. Three more generic tiers are shown: assemblies, subassemblies and components. These tiers illustrate the decomposition of a complex system into a series of fundamental physical problems. The number of tiers needed to compose a complete problem may be more or less than that shown. Figure 1: Validation hierarchy illustrating the decomposition of a full system model into submodels (components, subassemblies, and assemblies) UNCLASSIFIED/UNLIMITED UNCLASSIFIED/UNLIMITED Model Validation and Uncertainty Quantification Applied to Cervical Spine Injury Assessment 26 4 RTO-MP-AVT-147 In Figure 1, component problems are typically physics-based and represent important problem characteristics that the model must be able to simulate accurately. Examples of unit problems include material coupon tests, interface or joint tests, and load environment tests. Component problems will typically involve simplifications involving idealized geometry, boundary conditions and applied loads. Also, some component problems are required to determine fundamental constitutive properties; i.e., they represent calibrations (or parameter estimation, but not validation) against the experimental data. Careful construction o

Using physics-based models to predict the performance of engineered systems is becoming routine and is increasingly relied upon as a means to predict reliability when testing is prohibitive. To predict reliability, uncertainties in the system parameters must be modeled and propagated through the performance model using an appropriate probabilistic method. Uncertainties in engineered systems exist in loadings, environment, material strength, geometry, and manufacturing/assembly conditions. In many cases, these uncertainties are not direct physics-based model parameters. For example, the variations in the torque of a nut during assembly may be modeled as an initial penetration between two parts of the finite element model. Therefore, intermediate relationships between the physical uncertainties to the physics-based model are required. To account for these variations in the finite element model then requires a change to multiple nodal coordinates. Because of the significant time required to make these changes, a practical approach is required to model the geometry changes in complex finite element models. New capabilities in the NESSUS probabilistic analysis software for creating and applying shape vectors to geometry changes have been developed and implemented and are described in the paper. The probability density functions used to model the uncertainties in these parameters are ideally developed using experimental data or expert judgment. This paper describes several uncertainty modeling approaches for an actual probabilistic analysis using a non-linear transient finite element model in excess of 1 million elements. Examples of combining computational models, analytical equations, and experimental results are presented to relate computational model inputs in terms of measurable random variables.

On Feb 1, 2003, the Shuttle Columbia was lost during its return to Earth. As a result of the conclusion that debris impact caused the damage to the left wing of the Columbia Space Shuttle Vehicle (SSV) during ascent, the Columbia Accident Investigation Board recommended that an assessment be performed of the debris environment experienced by the SSV during ascent. A flight rationale based on probabilistic assessment is used for the SSV return-to-flight. The assessment entails identifying all potential debris sources, their probable geometric and aerodynamic characteristics, and their potential for impacting and damaging critical Shuttle components. A probabilistic analysis tool, based on the SwRI-developed NESSUS probabilistic analysis software, predicts the probability of impact and damage to the space shuttle wing leading edge and thermal protection system components. Among other parameters, the likelihood of unacceptable damage depends on the time of release (Mach number of the orbiter) and the divot mass as well as the impact velocity and impact angle. A typical result is visualized in the figures below. Probability of impact and damage, as well as the sensitivities thereof with respect to the distribution assumptions, can be computed and visualized at each point on the orbiter or summarized per wing panel or tile zone.

This paper presents an assessment case study on several segments of buried natural gas pipeline constructed in 1936 with ‘bell-bell-chill ring’ (BBCR) style girth weld joints, and currently operating in a seismically active region of North America. Seismic vulnerability was evaluated in terms of girth weld fracture and plastic collapse probabilities for specified hazards of varying severity and likelihood. Monte Carlo simulations performed in NESSUS® provided failure probability estimates from distributed inputs based on PIPLIN deformation analyses, nondestructive and destructive flaw sizing, residual stress measurements, weld metal tensile and CTOD tests, and limit state functions based on published stress intensity and collapse solutions.

The use of computational simulation is increasingly relied upon as performance requirements for engineered systems increase and as a means of reducing testing. Model verification and validation (V&V) provides a mechanism to develop computational models that are utilized for engineering predictions and ensure decisions with quantified confidence. The Los Alamos National Laboratory Dynamic Experimentation (DynEx) program is designing and validating steel blast containment vessels using limited experiments coupled with computational models. This paper describes the verification and validation of an analytical and computational model used to predict the penetration depth of explosively released fragments into the containment vessel structure. A systematic approach of model V&V is used to compare model predictions and experiments and establish metrics to quantify confidence. The use of uncertainty quantification is an essential part of V&V as there are inherent and subjective uncertainties in the model that must be correlated with the uncertainties from the experiments.

As a result of the conclusion that debris impact caused the damage to the left wing of the Columbia Space Shuttle Launch Vehicle (SSLV) during ascent, the Columbia Accident Investigation Board (CAIB) recommended that an assessment be performed of the complete debris environment experienced by SSLV during ascent. Eliminating the possibility of debris transport is not possible; therefore, a flight rationale based on probabilistic assessment is required for the SSLV return-to-flight (RTF). The assessment entails identifying all potential debris sources, their probable geometric and aerodynamic characteristics, and their potential for inflicting damage to the SSLV. This paper describes the development and verification of a probabilistic debris transport analysis (DTA) procedure.

As a result of the conclusion that debris impact caused the damage to the left wing of the Columbia Space Shuttle Launch Vehicle (SSLV) during ascent, the Columbia Accident Investigation Board (CAIB) recommended that an assessment be performed of the complete debris environment experienced by SSLV during ascent. Eliminating the possibility of debris transport is not possible; therefore, a flight rationale based on probabilistic assessment is required for the SSLV return-to-flight (RTF). The assessment entails identifying all potential debris sources, their probable geometric and aerodynamic characteristics, and their potential for inflicting damage to the SSLV. This paper describes the development and verification of a probabilistic debris transport analysis (DTA) procedure.

Probabilistic analyses allow the effect of uncertainty in system parameters on predicted model performance measures to be determined. Furthermore, using performance functions to describe a failure event, the probability of failure can be quantified. The effect of three-dimensional prosthesis shape optimization on the probabilistic response and failure probability of a cemented hip prosthesis system is investigated. Random variables include joint and muscle loading, cortical and cancellous bone and PMMA bone cement elastic properties, and strength parameters describing failure of the bone cement and the prosthesis-bone cement interface. Several performance functions describing the bone cement and prosthesis-cement interface are used to compute the probability of failure. When evaluated deterministically, most performance functions indicated a safe design, with the exception of interface tensile failure. However, when evaluated probabilistically, finite probabilities of failure were computed, some significant. The most likely mode of failure before shape optimization was prosthesis-bone cement interface tensile failure with a predicted probability of failure of 97.9%. Deterministic prosthesis shape optimization reduced the probability of failure for all performance functions and reduced prosthesis-bone cement interface tensile failure by 31.7%. Probability sensitivity factors indicate that the uncertainty in the joint loading, cement strength, and implant-cement interface strength have the greatest effect on the computed probability of failure. Implant shape optimization results in a more robust implant design that is less sensitive to uncertainties in joint loading, which cannot be easily controlled, and more sensitive to cement and interface properties, which are easier to modify.

Model verification and validation (V&V) provides a mechanism to develop computational models that can be used to make engineering predictions and decisions with quantified confidence. Model V&V procedures are needed to reduce the time, cost and risk associated with component and full-scale testing of products, materials and engineered systems. The Los Alamos National Laboratory Dynamic Experimentation (DynEx) program is designing and validating steel blast containment vessels using limited experiments coupled with computational models. This paper describes the testing program for the validation experiments in support of a verification and validation process for an analytical and computational model used to predict the penetration depth of explosively released fragments into the containment vessel structure. The V&V process is described as well as pre-test analytic modeling and validation experiments. Uncertainties in the experiments that may influence model validation are discussed from an uncertainty quantification perspective since there are inherent and subjective uncertainties in the model that must be correlated with the uncertainties from the experiments.

In May of 2002, three cracks were discovered in the flowliner of one of the orbiters near the interface with the low-pressure turbo pump. Cracking was identified as high-cycle fatigue due to flow-induced vibrations produced by the turbo pumps. Initial deterministic fracture mechanics analyses suggested that the fatigue cracks might lead to failure in a single flight. This result was believed to be conservative because of the inherent uncertainty in much of the input, which led to multiple worst-case assumptions in the deterministic analysis. The goal of the present work was to perform a probabilistic fracture mechanics analysis that explicitly accounted for the uncertainty in the input variables, rather than use the worst-case assumptions employed in prior deterministic analyses. Statistical models based on available data for key input variables to the analysis were developed to describe the uncertainty in dynamic loading and fatigue crack growth properties at cryogenic temperatures. New weight function solutions were also developed to more accurately describe the stress intensity factors driving the crack. Computational simulations using the Monte Carlo technique were used to establish the flowliner probability of failure (POF) per flight. A range of probability-of-detection (POD) curves was assumed for inspection of the flowliners since data were not available on this input variable at the time of the analysis. Results showed that relatively low POFs in the flowliner (0.001 to 0.0001) could be achieved provided cracks are detected using pre-flight inspections having adequate probability of detection (20 mil cracks with 50% POD, and 75 mil cracks with 99% POD). Thus, follow-up measurements to confirm the POD curves assumed in the analysis for pre-flight inspection are crucial. A flight rationale was recommended based on comparing the computed flowliner POFs with previously estimated POFs for engine failure and overall mission failure. A probabilistic sensitivity analysis showed that uncertainties in predicted flowliner fatigue failures are predominantly controlled by uncertainties in the values of the mean and standard deviation of the dynamic stress amplitudes. Based on these results recommendations are also provided on how best to increase the reliability of the orbiters LH2 feedline flowliner.

The Los Alamos National Laboratory Dynamic Experimentation (DynEx) program is the designing and validating steel blast containment vessels using limited experiments coupled with computational models. Through a need to design portions of the vessel to protect against breeches by projectiles, an analytical model was developed along the lines of the Walker–Anderson penetration model to predict the penetration depth of a projectile in a two- and three-layer target. The three-layer target consists of boron carbide ceramic (B4C), beryllium (Be), and aluminum. The two-layer target removes the Be. This model was integrated in the NESSUS® probabilistic analysis program to provide a deterministic and probabilistic design tool. Through a verification and validation approach, the model predictions are compared to the experimental results for both target configurations. The probabilistic analysis or uncertainty quantification is an essential part of verification and validation (V&V) and is used to provide confidence in model predictions. Overall, the V&V procedure indicates that the model predicts the two-layer target results well and is biased conservatively. The three-layer target provides reasonable predictions for thinner ceramic layers. The probabilistic results provide additional insight into the model and experimental results comparison over a deterministic analysis alone. The results show that there may be incomplete physics in the modeling of Be and thicker B4C layers. The probabilistic sensitivity factors show that the projectile density, velocity and strength, and Be strength are important variables. This information provides insight into approaches to improve the model predictions and establishes validity for use of the current model for specific configuration ranges.