Deep neural networks virtually dominate the domain of most modern vision systems, providing high performance at a cost of increased computational complexity. Since for those systems it is often required to operate both in real-time and with minimal energy consumption (e.g., for wearable devices or autonomous vehicles, edge Internet of Things (IoT), sensor networks), various network optimisation techniques are used, e.g., quantisation, pruning, or dedicated lightweight architectures. Due to the logarithmic distribution of weights in neural network layers, a method providing high performance with significant reduction in computational precision (for 4-bit weights and less) is the Power-of-Two (PoT) quantisation (and therefore also with a logarithmic distribution). This method introduces additional possibilities of replacing the typical for neural networks Multiply and ACcumulate (MAC—performing, e.g., convolution operations) units, with more energy-efficient Bitshift and ACcumulate (BAC). In this paper, we show that a hardware neural network accelerator with PoT weights implemented on the Zynq UltraScale + MPSoC ZCU104 SoC FPGA can be at least 1.4x more energy efficient than the uniform quantisation version. To further reduce the actual power requirement by omitting part of the computation for zero weights, we also propose a new pruning method adapted to logarithmic quantisation.
Modern digital Role Playing Games are often played on some sort of a map. Creating those maps by artists can take a long time. In this paper we propose a solution for automatic creation of such maps, that can then be used as a support tool for creating a baseline map to work on by artists, or as an automatic map generator. The general approach of creating maps using our solution is briefly explained, and then a closer look at the ground part of the map creation process is taken. Our algorithm takes as an input a list of tiles to be used and a rough sketch of a map to be generated, prepared with specific colours. It then creates a tile grid from that sketch using simple downscaling. That tile grid is then corrected using methods proposed in this paper, so that it does not have sharp edges and mismatched tiles. Finally, using tiles from input, the grid is transformed into an output map that closely resembles the sketch in its structure. Our solution is tested in practice by means of several exemplary sketches, each accompanied with comment explaining the significance of obtained result.
3D object detection from LiDAR sensor data is an important topic in the context of autonomous cars and drones. In this paper, we present the results of experiments on the impact of backbone selection of a deep convolutional neural network on detection accuracy and computation speed. We chose the PointPillars network, which is characterised by a simple architecture, high speed, and modularity that allows for easy expansion. During the experiments, we paid particular attention to the change in detection efficiency (measured by the mAP metric) and the total number of multiply-addition operations needed to process one point cloud. We tested 10 different convolutional neural network architectures that are widely used in image-based detection problems. For a backbone like MobilenetV1, we obtained an almost 4x speedup at the cost of a 1.13% decrease in mAP. On the other hand, for CSPDarknet we got an acceleration of more than 1.5x at an increase in mAP of 0.33%. We have thus demonstrated that it is possible to significantly speed up a 3D object detector in LiDAR point clouds with a small decrease in detection efficiency. This result can be used when PointPillars or similar algorithms are implemented in embedded systems, including SoC FPGAs. The code is available at https://github.com/vision-agh/pointpillars\_backbone.
A method of human skeleton-tracking and -refinement, and feature extraction for two-person interaction recognition in video is proposed. Its purpose is to properly reassign the same person-representing skeletons, approximate the missing joints and extract meaningful relational features. In addition, based on the created feature streams, two different multi-stream deep neural networks are designed to perform data transformation and interaction classification. They provide different relations between model complexity and performance quality. The first one is an ensemble of "weak" pose-based action classifiers, which are trained on different time-phases of an interaction. At the same time, the overall classification result is a time-driven aggregation of weighted combinations of their results. In the second approach, three input feature streams were created, which fed a triple-stream LSTM network. Both network models were trained and tested on the interaction subset of the NTU RGB+D data set, showing comparable performance with the best reported CNN- and Graphic CNN-based classifiers.
In recent years Human Object Interaction (HOI) detection has experienced rapid performance growth mainly due to the development of various deep learning-based methods and algorithms. One of the most popular approaches to this task is three-stream architecture consisting of three processing paths: human-stream, object-stream, and relation-stream, gathering information on the scene using features extracted from particular image elements. The total amount of features depends on the detailed processing schemes and model parameters. The increase in the number of features does not necessarily increase the final efficiency of the HOI detection method. This paper focuses on the relation between the quantity of features used to detect HOI and its predictive efficiency. In our study, we investigate the influence of the number of features on the final detection results. Several experiments were conducted to validate various model configurations of different features considered.
In the paper, a method for image search is proposed. It allows for finding images starting from a text query that contains names of object classes and their expected spatial relations. The input query containing the description of desired objects and their spatial relations is used to select and score relevant images. Next, the score is used to sort the output images putting more relevant ones on the top of the list. The proposed approach is based on object detection methods and a fuzzy approach describing the position of detected objects in relation to other ones. The method may be used to retrieve images from databases containing images with metadata produced by object detectors.
Triangle strips provide a memory-efficient representation of triangle meshes. In the ideal case, they allow to encode a mesh of n faces with $$n + 2$$ vertices. All modern graphics interfaces offer a primitive type for triangle strips to disburden the host/device bottleneck and save graphics memory. Although modern 3D rendering pipelines mainly utilize triangle lists, strips still offer significant benefits in two-dimensional applications, e.g. the visualization of polygons in orthographic maps. Encoding a regular mesh into an optimal triangle strip representation is an NP-complete problem. Therefore, multiple heuristic techniques for different cost metrics have been proposed and implemented. In this paper, we provide an overview of related approaches and propose a unified cost model to rate their quality. Furthermore, we introduce a faster and more flexible implementation for triangle strip generation. It extends an established key technique, the tunneling operator, with a new algorithm for circle detection. A concluding benchmark compares our code with existing solutions and outlines its superior performance in most use cases.
Analysis of errors in two methods of color visual cryptography with random shares introduced by us in previous publications was performed. In both methods the shares are random, and consequently, errors occur in the decoded image. In one of the methods, where the coding is done by unhiding the pixels, there are two types of errors: missing color errors and hiding failure errors. In the other method, where the coding is done by unhiding, only the missing color errors occur. Probabilities of the missing color errors were modelled mathematically and frequencies of two types of errors were tested experimentally for both methods. Tests demonstrate that the model is correct. The results show in which cases the considered methods exhibit more accurate decoding results. Also, the extended results of randomness tests conducted with the NIST statistical test suite on the results of coding for a set of typical benchmark images are presented in the form of histograms of p-values. These graphical results indicate that the shares are indeed truly random.
In this paper, research on the influence of parameters' values in the pipelines of facial-based reidentification systems is presented. It was assumed that the solution should operate in real time in conditions typical of the reidentification system to be used. Such conditions were obtained as part of research regarding the reidentification of aggressively acting people during sports events. Typically, such a pipeline consists of many steps, including facial region detection, frontalisation, embedding, and classification, which are usually evaluated separately. This paper focuses on the parameters of facial alignment and classification in the context of systems based on well-established solutions of Multi-task Cascaded Convolutional Networks coupled with Inception Resnet embedding. The authors propose evaluating the results of the entire pipeline as a way to identify the optimal set of parameters for each step, thus producing a pipeline where the subsequent steps are best fitted to each other rather than giving the best results on their own. The results indicate that the correct selection of parameters of the steps of the pipeline depends on further steps used and vice versa. It is therefore suboptimal to select parameters based on a separately evaluated set of steps, as it is usually presented in the literature. The reidentification pipeline must therefore be evaluated as a whole, disregarding the results achieved by any single part of the pipeline, as they are not an indicator of overall system performance.
The wah-wah is a guitar effect used to modulate the sound while playing. This is an unusual effect in that the guitar player, having his hands on instrument, controls it in real time with the foot. The digital equivalent proposed in this paper transfers this control to mouth movements by capturing an image from a computer camera and then applying computer vision algorithms. The paper analyzes the applicability and studies the effectiveness of using mouth movement to control a wah-wah type guitar effect.
Many systems which check queue length at intersections are inaccurate. Information about small/medium/large queues are not enough for intelligent transport systems which could modify an actual intersection program in a way which may allow for optimising the traffic flow in the city. Not only does the presented system provide information about queue length in metres on every lane, but also it is based solely on camera focal length, sensor size and number of lanes in camera view, which minimizes the involvement of traffic operators in time-consuming camera setup. The system consists of several submodules, the first of which detects license plates and uses them to create a configuration of the camera. Subsequently, the second module that performs detection of the type of vehicle can determine what the current length of the queue on every lane is. All this information is sent to the traffic management system in Wroclaw which modifies the traffic lights controller programs, optimising the traffic flow in the city.
In crowd analysis tasks (crowds of humans, cattle, birds, drones, etc.) the low-level vision tools are usually the same, i.e. detection and tracking of either individuals or groups. The required results, however, are more complicated (e.g. patterns of group splitting/merging, changes in group sizes and membership, group formation and disappearance, etc.). To complete such tasks, raw results of detection/tracking are converted into data associations representing crowd structure/evolution. Normally, those associations are deterministic and based on target labeling. However, performances of detectors/trackers are non-perfect, i.e. their outcomes are effectively non-deterministic. We discuss matrix-based mathematical models of interactions between detectors and trackers to represent such data associations non-deterministically. In particular, a methodology for reconstructing weak or missing associations by alternative sequences of matrix operations is proposed. This can provide more reliable label correspondences between selected moments/points of monitored scenes. Apart from mathematical details, the paper presents examples illustrating feasibility of the proposed approach.
We describe experiments with the hi-tech contactless scanner CRUSE CS 220ST1100 in the digitization of originals of natural and cultural heritage. The 2D scans guarantee high accuracy both in geometry and radiometry (48 bits for RGB colors). However, an inexperienced customer needs support in selecting the appropriate scan mode. To distinguish similar CRUSE scans, we proposed an image descriptor based on Harris corners and a topological structure embedding a planar subgraph. For some use-cases, the Harris approach did not perform well. We report on a novel SIFT type detector using concurrent color channels, hence the proposed name. We put our solution into the context of previous research and compare, on selected use-cases, the solution quality and/or disadvantages.
In the paper, an application of the recently proposed approach to hybrid image binarization based on pixel voting is considered for industrial images. Since such images typically contain the text embossed or engraved in metal nameplates, often non-uniformly illuminated, a proper binarization of such images is usually much harder than for scanned document images, or even for the photos of text documents. Assuming that no single method would be the best solution for such images, a hybrid solution, based on the combination of multiple algorithms using pixel voting, has been recently proposed for document images. The obtained experimental results for the dataset of "industrial" images confirm the usefulness of this approach and the proposed combinations of previously developed algorithms outperform the other methods, making it possible to increase the OCR accuracy also for demanding images containing light reflections and shadows.
Colour images tend to be more visually appealing to humans compared to grayscale images, as colour images are closer in representation to the natural way we perceive our environment. While obtaining grayscale images from colour images is relatively trivial, the reverse process is not. In this paper, a machine learning method inspired by the Bayer filter and the demosaicing process of digital colour cameras, is proposed for the colourisation of grayscale images. The method involves training a multilayer perceptron model on colour images that are semantically similar to each other. The model can, henceforth, colourise grayscale images that are semantically similar to those in the training set. The success of our method is dependent on an image data representation model developed for this purpose. The proposed model gives impressive results despite requiring no human intervention and less machine resources for training when compared with existing deep learning models.
Quantum Neural Networks (QNNs) are an emerging technology that can be used in many applications including computer vision. In this paper, we presented a traffic sign classification system implemented using a hybrid quantum-classical convolutional neural network. Experiments on the German Traffic Sign Recognition Benchmark dataset indicate that currently QNN do not outperform classical DCNN (Deep Convolutuional Neural Networks), yet still provide an accuracy of over 90% and are a definitely promising solution for advanced computer vision.
The objective of this study is the segmentation of the intima-media complex of the common carotid artery, on longitudinal ultrasound images, to measure its thickness. We propose a fully automatic region-based segmentation method, involving a supervised region-based deep-learning approach based on a dilated U-net network. It was trained and evaluated using a 5-fold cross-validation on a multicenter database composed of 2176 images annotated by two experts. The resulting mean absolute difference (<120 um) compared to reference annotations was less than the inter-observer variability (180 um). With a 98.7% success rate, i.e., only 1.3% cases requiring manual correction, the proposed method has been shown to be robust and thus may be recommended for use in clinical practice.
Histopathological sections allow pathologists to evaluate a wide range of specimens, including breast cancer, obtained from biopsies and surgical procedures. The accuracy of the employed automated cell detection technique is critical in obtaining efficient diagnostic performance. In this paper we investigate 18 different adaptive threshold methods based on various approaches. We validate the methods on a set of histopathological images of breast cancer, where immunohistochemical staining of FOXP3 was performed with 3,3’-diaminobenzidine and hematoxylin. The thresholding is performed on monochromatic images derived from original images: separate channels of Red-Green-Blue and Hue-Saturation-Value, layers of results of color deconvolution, ‘brown’ channel, ‘blue-ratio’ layer. The main objective of the evaluation is to determine if the detected objects obtained by the tested methods of thresholding are consistent with the manually labeled ones. The performance is evaluated using precision, sensitivity and F1 score measures. It appears that satisfactory results were achieved only by 6 methods. It was found that bradley method is the best performing method for nuclei detection in this type of stained tissue samples. It has best sensitivity value for images after color deconvolution and Value layer (of Hue-Saturation-Value color space), 0.970 and 0.975 respectively. As a result, we recommend a most efficient local threshold technique in the case of nuclei detection in digitized immunohistochemically stained tissue sections. This initial detection of objects followed by texture, size and shape analysis will give a collection of cells’ nuclei to perform further accurate segmentation. The proposed detection method will be used in a framework focused on computer-aided diagnosis.
In this paper the research on optimisation of visual object tracking using a Siamese neural network for embedded vision systems is presented. It was assumed that the solution shall operate in real-time, preferably for a high resolution video stream, with the lowest possible energy consumption. To meet these requirements, techniques such as the reduction of computational precision and pruning were considered. Brevitas, a tool dedicated for optimisation and quantisation of neural networks for FPGA implementation, was used. A number of training scenarios were tested with varying levels of optimisations-from integer uniform quantisation with 16 bits to ternary and binary networks. Next, the influence of these optimisations on the tracking performance was evaluated. It was possible to reduce the size of the convolutional filters up to 10 times in relation to the original network. The obtained results indicate that using quantisation can significantly reduce the memory and computational complexity of the proposed network while still enabling precise tracking, thus allow to use it in embedded vision systems. Moreover , quantisation of weights positively affects the network training by decreasing overfitting.
Facial age estimation studies have shown that the type of features used for face representation significantly impact the accuracy of age estimates. This work proposes a novel method of representing the face with compact facial features derived from extracted raw image pixels and Local Binary Patterns (LBP) for age estimation. The compact facial features are realized by exploiting the statistical properties of extracted facial features while aggregating over a whole feature set. The resulting compact feature set is of reduced dimensionality compared to the noncompact features. It also proves to retain relevant facial features as it achieves better age prediction accuracy than the non-compact features. An age group ranking model is also proposed which further reduces facial features dimensionality while improving age estimation accuracy. Experiments on the publicly-available FG-NET and Lifespan datasets gave a Mean Absolute Error (MAE) of 1.76 years and 3.29 years respectively, which are the lowest MAE so far reported on both datasets, to the best of our knowledge.
