Within the recent years, the concept of Digital Twins (DT) emerged to support digital engineering physical-systems design in the processing and analysis of device components, data flows, and networked systems. As an element of systems engineering, the DT serves as a method for life cycle management for operations and maintenance through monitoring, diagnostics, and prognostics. Key to DT methods is the use of distributed sensors to monitor the system and determine the functioning with the use of physical design information, such as a series of distributed edge sensors and the layout of a physical electrical grid. While industries like industrial manufacturing, electrical power, space systems, and healthcare maintenance have embraced DT; other groups are utilizing the concept for analysis. Given that a large number of sensors are used to gather data on the health of system, it is natural that data fusion, estimation theory, and signals processing support digital twin fusion (DTwF); but there are a variety of challenges such as big data, distribution fusion, and edge analytics. The panel will discuss areas in which data and information fusion techniques enhance DT applications.

Artificial Intelligence/Deep Learning (AI/DL) techniques are based on learning a model using large available data sets. The data sets typically are from a single modality (e.g., imagery) and hence the model is based on a single modality. Likewise, multiple models are each built for a common scenario (e.g., video and natural language processing of text describing the situation). There are issues of robustness, efficiency, and explainability that need to be addressed. A second modality can improve efficiency (e.g., cueing), robustness (e.g., results cannot be fooled such as adversary systems), and explainability from different sources. The challenge is how to organize the data needed for joint data training and model building. For example, what is needed is (1) structure for indexing data as an object file, (2) recording of metadata for effective correlation, and (3) supporting methods of analysis for model interpretability for users. The Panel presents a variety of questions and responses discussed, explored, and analyzed for data fusion-based AI data fusion tools.

Motivation for the panel presentation was to describe a learning technique: (1) which is unlike a black box, (2) along the proposed principles of the DARPA XAI program [1], and (3) can provide explanatory/interpretable capability. The technique is based on the Maximum Entropy (MaxEnt) model -based machine learning approach applied to Distributed Decision Fusion (DDF) for both multimodal sensing [2] target detection and target classification/recognition. The approach was described in previous publications, and demonstrated the MaxEnt model -based machine learning DDF achieving the best, realistic detection perfounance demonstrating the effectiveness of the method. The same effectiveness is expected applying MaxEnt model -based machine learning to multimodal sensing for target classification/recognition as described in -part below and references. The model -based aspect is the key element of the algorithm enabling to provide explanatory/interpretable capability.

AI techniques are based on learning a model based on a large available data set. The data sets typically are from a single modality (e.g., imagery) and hence the model is based on a single modality. Likewise, multiple models are each built for a common scenario (e.g., video and natural language processing of text describing the situation). There are issues of robustness, efficiency, and explainability needed. A second modality can improve efficiency (e.g., cueing), robustness (e.g., results can not be fooled such as adversary systems), and explainability from different sources help. The challenge is how to organize the data needed for joint data training and model building. For example, what is needed (1) structure for indexing data as an object file, (2) recording of metadata for effective correlation, and (3) supporting models and analysis for model interpretability for users. There are a variety of questions to be discussed, explored, and analyzed for fusion-based AI tool.

During the 2019 SPIE DSS conference, panelists were invited to highlight the trends and use of artificial intelligence and machine learning (AI/ML) for information fusion. The common themes between the panelists include leveraging AI/ML coordinated with Information Fusion for: (1) knowledge reasoning, (2) model building, (3) object recognition and tracking, (4) multimodal learning, and (5) information processing. The opportunity for machine learning exists within all the fusion levels of the Data Fusion Information Group model.

In a prior paper, a probabilistic model for using context in fusion was developed. It was shown that context-based fusion could be represented by a Bayesian probabilistic model that contains situation and context data, as well as conditional probabilities for the random variables. In the same paper, a conceptual model of an adaptive real-time context management system was proposed to monitor fusion performance, and select the appropriate context in order to improve fusion performance. This paper represents an extension of the above paper by developing frameworks for an adaptive general real-time context management, with application to optimize the tracking performance of an airborne platform.

During the 2018 SPIE DSS conference, panelists were invited to highlight the trends and use of artificial intelligence and deep learning (AI/DL) for information fusion. This paper highlights the common issues presented from the panel discussion. The key issues include: leveraging AI/DL coordinated with information fusion for: (1) knowledge reasoning and reasoning, (2) information fusion enhancement, (3) object recognition and tracking, (4) data with models fusion, and (5) deep multimodal fusion cognition strategies to support the user.

The Maximum Entropy (MaxEnt) information theoretic model parametric framework was introduced in a prior paper for distributed decision fusion (DDF) without knowledge of prior probabilities of local decisions. The paper demonstrated the effectiveness of the MaxEnt fusion center by achieving the best, realistic detection performance with respect to published results of either the Bayesian formulation or the Neyman-Pearson criterion. This paper represents the framework of an extension of MaxEnt DDF, called E-MaxEnt using: individual sensor MaxEnt classifiers for targets classification/recognition, and by fusing local classifier decisions. Specifically, in E-MaxEnt each sensor has a front-end pre-processing system for both signal detection and to process unique target attributes extracted for example from observed target imagery, which attributes are stored for reference/learning/comparison in the sensors MaxEnt classifiers. Based on the degree of match, each sensor generates local binary decisions that are sent to a MaxEnt fusion center, in the usual parallel architecture. No assumptions are made about knowing any local decision rules. The sensors are taking simultaneous (synchronized) measurements with overlapping FOV overages. It should be noted that the above description is not meant to address the “needle-in-haystack” problem, but rather address finding the presence, viz., classify/recognize a previously seen “known” target in areas where previously seen targets most likely are, along with other targets. At the time of writing, the data sets to test the algorithm were not available, but front-end image processing and MaxEnt classifiers were implemented. It is hoped that someone could provide the necessary data sets so the efficacy of the method could be demonstrated and compared with alternative approaches.

Quantum physics has a growing influence on sensor technology; particularly, in the areas of quantum computer science, quantum communications, and quantum sensing based on recent insights from atomic, molecular and optical physics. These quantum contributions have the potential to impact information fusion techniques. Quantum information technology (QIT) methods of interest suggest benefits for information fusion, so a panel was organized to articulate methods of importance for the community. The panel discussion presented many ideas from which the leading impact for information fusion is directly related to the sub-Rayleigh sensing that reduces uncertainty for object assessment through enhanced resolution. The second areas of importance is in the cyber security of data that supports data, sensor, and information fusion. Some elements of QIT that require further analysis is in quantum computing for which only a limited set of information fusion techniques can harness the methods associated with quantum computer architectures. The panel reviewed various aspects of QIT for information fusion which provides a foundation to identify future alignment between quantum and information fusion techniques.

During the 2016 SPIE DSS conference, nine panelists were invited to highlight the trends and opportunities in cyber-physical systems (CPS) and Internet of Things (IoT) with information fusion. The world will be ubiquitously outfitted with many sensors to support our daily living thorough the Internet of Things (IoT), manage infrastructure developments with cyber-physical systems (CPS), as well as provide communication through networked information fusion technology over the internet (NIFTI). This paper summarizes the panel discussions on opportunities of information fusion to the growing trends in CPS and IoT. The summary includes the concepts and areas where information supports these CPS/IoT which includes situation awareness, transportation, and smart grids.

The concept of goal lattices has been presented as a method with which to determine the mission value of each admissible sensor action as an aid in computing the associated expected information value rate. Implicit collaboration of sensing platforms has previously been presented as a method for coordinating the actions of autonomous entities with shared goals. In this paper we formally define that interaction and show how joint mission goals common to collaborating agents can be used to evaluate admissible sensing actions. The net result of incorporating joint goals is expected to reduce the amount of redundant sensing actions when compared to autonomous sensing systems operating without joint, collaborating goals.

During the 2015 SPIE DSS conference, nine panelists were invited to highlight the trends and use of context for information fusion. This paper highlights the common issues and trends presented from the panel discussion. The different panelists highlighted methods of filtering methods, data aggregation, and the importance of context for realtime analytics. Using the panelist perspectives, the review organizes the common issues and themes as well areas of future analysis of content and context enrichment from information fusion.

We introduce the method of the Maximum Entropy (MaxEnt) model for fusing local decisions in a distributed multiple sensor system. The fusion center receives local binary decisions in the usual parallel architecture. No assumptions are made about knowing any local decision rules. Our approach is based on the concept of machine learning, wherein the MaxEnt parametric model is used for supervised classification and prediction serving as the central (global) decision rule. Therefore, the system is able to learn the detection performance of the sensors as a function of time without prior knowledge of the actual probabilities of local decisions, only requiring an initial set of random training data. Thus it is demonstrated that the system is adaptive and can learn contextual changes of the sensors. Furthermore, we provide simulation results comparing the MaxEnt fusion center performance with published results using both the Bayesian formulation and Neyman-Pearson criterion and with MaxEnt achieving the best, realistic detection performance demonstrating the effectiveness of the method.

The integration of hard (physical) and soft (meta-physical) contexts in an info- rmation fusion system requires the identification of the specific mission oriented goals which it is desired to achieve. Just as most sensors cannot acquire data omnidirectionally, it is not computationally feasible to evaluate all contexts within which acquired data can be understood by an information fusion system. We first define a notional problem consisting of operating and hiding areas and transit routes between them. We then define physical and meta-physical contexts within which data acquired from the observed area can be interpreted and define the piecewise application of context specific transforrnations to a partition of the global problem of understanding data in context.

We depict the functions and extension of a novel information theory based sensors management (ISBM) system to information based sensor and mission management (IBSMM) such that intelligence collection is effective in an expected value sense while remaining independent of any particular platform, sensor or point solution. We describe the proposed implementation via implicit collaboration through common mission goals. Integral to this concept is utilizing the scope of knowledge at each sensing resource in order to provide context sensitive information extraction from sensor data via context-based information fusion.

With the plethora of information, there are many aspects to contested environments such as the protection of information, network privacy, and restricted observational and entry access. In this paper, we review and contrast the perspectives of challenges and opportunities for future developments in contested environments. The ability to operate in a contested environment would aid societal operations for highly congested areas with limited bandwidth such as transportation, the lack of communication and observations after a natural disaster, or planning for situations in which freedom of movement is restricted. Different perspectives were presented, but common themes included (1) Domain: targets and sensors, (2) network: communications, control, and social networks, and (3) user: human interaction and analytics. The paper serves as a summary and organization of the panel discussion as towards future concerns for research needs in contested environments.

Over the last two decades, many solutions have arisen to combine target tracking estimation with classification methods. Target tracking includes developments from linear to non-linear and Gaussian to non-Gaussian processing. Pattern recognition includes detection, classification, recognition, and identification methods. Integrating tracking and pattern recognition has resulted in numerous approaches and this paper seeks to organize the various approaches. We discuss the terminology so as to have a common framework for various standards such as the NATO STANAG 4162 -Identification Data Combining Process. In a use case, we provide a comparative example highlighting that location information (as an example) with additional mission objectives from geographical, human, social, cultural, and behavioral modeling is needed to determine identification as classification alone does not allow determining identification or intent.

During the SPIE 2012 conference, panelists convened to discuss “Real world issues and challenges in Human Social/Cultural/Behavioral modeling with Applications to Information Fusion.” Each panelist presented their current trends and issues. The panel had agreement on advanced situation modeling, working with users for situation awareness and sense-making, and HSCB context modeling in focusing research activities. Each panelist added different perspectives based on the domain of interest such as physical, cyber, and social attacks from which estimates and projections can be forecasted. Also, additional techniques were addressed such as interest graphs, network modeling, and variable length Markov Models. This paper summarizes the panelists discussions to highlight the common themes and the related contrasting approaches to the domains in which HSCB applies to information fusion applications.