In decision support systems, information from many different sources must be integrated and interpreted to aid the process of gaining situational understanding. These systems assist users in making the right decisions, for example when under time pressure. In this work, we discuss a controlled automated support tool for gaining situational understanding, where multiple sources of information are integrated. In the domain of operational safety and security, available data is often limited and insufficient for sub-symbolic approaches such as neural networks. Experts generally have high level (symbolic) knowledge but may lack the ability to adapt and apply that knowledge to the current situation. In this work, we combine sub-symbolic information and technologies (machine learning) with symbolic knowledge and technologies (from experts or ontologies). This combination offers the potential to steer the interpretation of the little data available with the knowledge of the expert. We created a framework that consists of concepts and relations between those concepts, for which the exact relational importance is not necessarily specified. A machine-learning approach is used to determine the relations that fit the available data. The use of symbolic concepts allows for properties such as explainability and controllability. The framework was tested with expert rules on an attribute dataset of vehicles. The performance with incomplete inputs or smaller training sets was compared to a traditional fully-connected neural network. The results show it as a viable alternative when data is limited or incomplete, and that more semantic meaning can be extracted from the activations of concepts.

In this day and age, there exists an increasing need for systems and architectures able to process spatio-temporal data in a timely way. As a result, this paper presents CEP-traj, a novel middleware to ease the development of real-time trajectory-based services based on the Complex Event Processing (CEP) paradigm. By means of an event-based approach, the present middleware is able to detect a set of generic patterns along with meaningful changes of an entity s movement. In order to prove its suitability and feasibility, a vessel abnormal-behaviour detection system has been developed on the basis of the middleware s features. Finally, both synthetic and real datasets have been used to test the accuracy and performance of the middleware and the detection system implemented on top of the Esper engine. HighlightsA novel palette of datasets for CEP-traj and VAbDS evaluation.Evaluation of the middleware with network-constrained trajectories.Comparative of CEP-traj and a well-established online trajectory-processing framework.Evaluation of VAbDS by using a public real-world dataset.

Threat detection is a challenging problem, because threats appear in many variations and differences to normal behaviour can be very subtle. In this paper, we consider threats on a parking lot, where theft of a truck's cargo occurs. The threats range from explicit, e.g. a person attacking the truck driver, to implicit, e.g. somebody loitering and then fiddling with the exterior of the truck in order to open it. Our goal is a system that is able to recognize a threat instantaneously as they develop. Typical observables of the threats are a person's activity, presence in a particular zone and the trajectory. The novelty of this paper is an encoding of these threat observables in a semantic, intermediate-level representation, based on low-level visual features that have no intrinsic semantic meaning themselves. The aim of this representation was to bridge the semantic gap between the low-level tracks and motion and the higher-level notion of threats. In our experiments, we demonstrate that our semantic representation is more descriptive for threat detection than directly using low-level features. We find that a person's activities are the most important elements of this semantic representation, followed by the person's trajectory. The proposed threat detection system is very accurate: 96.6 % of the tracks are correctly interpreted, when considering the temporal context.

Maritime situation awareness is supported by a combination of satellite, airborne, and terrestrial sensor systems. This paper presents several solutions to process that sensor data into information that supports operator decisions. Examples are vessel detection algorithms based on multispectral image techniques in combination with background subtraction, feature extraction techniques that estimate the vessel length to support vessel classification, and data fusion techniques to combine image based information, detections from coastal radar, and reports from cooperative systems such as (satellite) AIS. Other processing solutions include persistent tracking techniques that go beyond kinematic tracking, and include environmental information from navigation charts, and if available, ELINT reports. And finally rule-based and statistical solutions for the behavioural analysis of anomalous vessels. With that, trends and future work will be presented.

Ground surveillance is normally performed by human assets, since it requires visual intelligence. However, especially for military operations, this can be dangerous and is very resource intensive. Therefore, unmanned autonomous visualintelligence systems are desired. In this paper, we present an improved system that can recognize actions of a human and interactions between multiple humans. Central to the new system is our agent-based architecture. The system is trained on thousands of videos and evaluated on realistic persistent surveillance data in the DARPA Mind’s Eye program, with hours of videos of challenging scenes. The results show that our system is able to track the people, detect and localize events, and discriminate between different behaviors, and it performs 3.4 times better than our previous system.

The contribution of this paper is a search engine that recognizes and describes 48 human actions in realistic videos. The core algorithms have been published recently, from the early visual processing (Bouma, 2012), discriminative recognition (Burghouts, 2012) and textual description (Hankmann, 2012) of 48 human actions. We summarize the key algorithms and specify their performance. The novelty of this paper is that we integrate these algorithms into a search engine. In this paper, we add an algorithm that finds the relevant spatio-temporal regions in the video, which is the input for the early visual processing. As a result, meta-data is produced by the recognition and description algorithms. The meta-data is filtered by a novel algorithm that selects only the most informative parts of the video. We demonstrate the power of our search engine by retrieving relevant parts of the video based on three different queries. The search results indicate where specific events occurred, and which actors and objects were involved. We show that events can be successfully retrieved and inspected by usage of the proposed search engine.

Presented is a hybrid method to generate textual descriptions of video based on actions. The method includes an action classifier and a description generator. The aim for the action classifier is to detect and classify the actions in the video, such that they can be used as verbs for the description generator. The aim of the description generator is (1) to find the actors (objects or persons) in the video and connect these correctly to the verbs, such that these represent the subject, and direct and indirect objects, and (2) to generate a sentence based on the verb, subject, and direct and indirect objects. The novelty of our method is that we exploit the discriminative power of a bag-of-features action detector with the generative power of a rule-based action descriptor. Shown is that this approach outperforms a homogeneous setup with the rule-based action detector and action descriptor.

Surveillance is normally performed by humans, since it requires visual intelligence. However, this can be dull and dangerous, especially for military operations. Therefore, unmanned autonomous visual-intelligence systems are desired. In this paper, we present a novel system that can recognize human actions, which are relevant to detect operationally significant activity. Central to the system is a break-down of high-level perceptual concepts (verbs) in simpler observable events. The system is trained on 3482 videos and evaluated on 2589 videos from the DARPA Mind's Eye program, with for each video human annotations indicating the presence or absence of 48 different actions. The results show that our system reaches good performance approaching the human average response.

An approach for decision support, consisting of situation and threat assessment is described, as part of a distributed and adaptive multi-sensor fusion engine. The decision support module combines the management and assessment components for situations and threats. The situation assessment determines relations between observed and geographical objects. Threat assessment relates the situations and other object information to threats. The threat assessment approach is based on a rule based expert system, implemented in one or more Bayesian networks. Results are presented for an example scenario in an urban environment.

An architecture for a distributed and adaptive multi-sensor fusion engine is described. The engine is developed to improve the protection of European armed forces against the threats troops are facing in urban environments. In particular, the engine needs to be scalable, flexible to quickly adjust to different missions and mission goals, easily extendable and suited for distributed implementation. The architecture follows a layered approach to define the main information processing elements.

In present-day military security operations threats are more difficult to reveal than in conventional warfare theatres, since they take place during the course of normal life. These maritime missions often take place in littoral environments, where acts of piracy, drug trafficking and other threatening events become obscured in the crowd of everyday fisheries, cargo traders, ferries and pleasure cruises, hindering situation awareness. We aim to improve situation awareness and threat detection capabilities in maritime scenarios by combining sensor-based information with context information and intelligence from various sources. The fusion and analysis in order to reveal suspect from normal behavior is based on domain ontologies. A test bed allows the study of various exploitation and assessments techniques applied to these domain ontologies. Using an appropriate scenario we have simulated suspect and normal behaviour to test the applicability of the various techniques.

Because of global economic and socio-political changes, an increase of conflicts near the world's coastlines is anticipated. The littoral zone is characterized by intense regular vessel traffic. The conduct of Maritime Security Operations and Peace support Operations therefore means that navies have to control instead of dominate the sea, allowing regular vessel traffic in the area of operations, and act against irregular adversaries who nevertheless also can possess military armaments. Piracy, drug trafficking and other threatening events become obscured in the crowd of everyday fisheries, cargo traders, ferries and pleasure cruises, hindering the detection of anomalies and suspect behavior, and resulting in insufficient situation awareness. For controlling these situations information superiority and adequate situation awareness is a necessity. For the purpose to achieve information superiority a research program at TNO, in collaboration with the RNLN, has started aiming at improving maritime situation awareness. To improve situation awareness and threat detection capabilities in maritime scenarios the combination of sensor-based information with context information and intelligence from various sources is required. In the study the fusion and analysis for revealing anomalies and suspect from normal behavior are based on domain ontologies. A test bed allows the study of various exploitation and assessments techniques applied to these domain ontologies. Using an appropri- ate scenario we have simulated suspect and normal behaviour to test the applicability of these techniques.

To effectively protect against threats on compounds or other forms of bases, decisions have to be made about follow-up actions. Without appropriate situation awareness threats cannot be timely recognised, so that decisions to contribute to the level of protection cannot be made in time. For acquiring this situation awareness the environment has to be measured with sensors and information about the threats should be processed as fast as possible. This paper will focus on the process to achieve the necessary situation awareness for the compound as an integrated platform with the aim of making effective protection possible. It is a report on results from the Dutch research program ‘Protection and Survivability of Compounds’.