This paper presents LiFT, a lightweight, fully quantized 3D object detection algorithm for LiDAR data, optimized for real-time inference on FPGA platforms. Through an in-depth analysis of FPGA-specific limitations, we identify a set of FPGA-induced constraints that shape the algorithm’s design. These include a computational complexity limit of 30 GMACs (billion multiply-accumulate operations), INT8 quantization for weights and activations, 2D cell-based processing instead of 3D voxels, and minimal use of skip connections. To meet these constraints while maximizing performance, LiFT combines novel mechanisms with state-of-the-art techniques such as reparameterizable convolutions and fully sparse architecture. Key innovations include the Dual-bound Pillar Feature Net, which boosts performance without increasing complexity, and an efficient scheme for INT8 quantization of input features. With a computational cost of just 20.73 GMACs, LiFT stands out as one of the few algorithms targeting minimal-complexity 3D object detection. Among comparable methods, LiFT ranks first, achieving an mAP of 51.84 https://github.com/vision-agh/lift .

For perception systems in mobile robotics - particularly in demanding, highly dynamic environments - event cameras (DVS - Dynamic Vision Sensors) are being increasingly utilised as an alternative to conventional vision sensors. The real-time processing of the registered spatio-temporal sparse point cloud must satisfy stringent requirements in terms of latency, throughput, and energy efficiency. In this demonstration, we present our method for implementing Graph Convolutional Neural Networks on a heterogeneous SoC FPGA platform, aimed at meeting these constraints. We address the challenges associated with integrating event-based sensors with reconfigurable hardware, as well as the intricate relationship between latency, performance, and hardware resource utilisation.

The performance of object detection systems in automotive solutions must be as high as possible, with minimal response time and, due to the often battery-powered operation, low energy consumption. When designing such solutions, we therefore face challenges typical for embedded vision systems: the problem of fitting algorithms of high memory and computational complexity into small low-power devices. In this paper we propose PowerYOLO - a mixed precision solution, which targets three essential elements of such application. First, we propose a system based on a Dynamic Vision Sensor (DVS), a novel sensor, that offers low power requirements and operates well in conditions with variable illumination. It is these features that may make event cameras a preferential choice over frame cameras in some applications. Second, to ensure high accuracy and low memory and computational complexity, we propose to use 4-bit width Powers-of-Two (PoT) quantisation for convolution weights of the YOLO detector, with all other parameters quantised linearly. Finally, we embrace from PoT scheme and replace multiplication with bit-shifting to increase the efficiency of hardware acceleration of such solution, with a special convolution-batch normalisation fusion scheme. The use of specific sensor with PoT quantisation and special batch normalisation fusion leads to a unique system with almost 8x reduction in memory complexity and vast computational simplifications, with relation to a standard approach. This efficient system achieves high accuracy of mAP 0.301 on the GEN1 DVS dataset, marking the new state-of-the-art for such compressed model.

Streszczenie: Systemy wizyjne to dynamicznie rozwijająca

In this demo, we present a hardware-software system for 3D object detection in LiDAR point clouds based on a deep neural network. The PointPillars architecture was used in this research, as it is a reasonable compromise between detection accuracy and computational complexity. The Brevitas tool was used for network quantisation and the FINN tool for hardware-software implementation in the reprogrammable Zynq UltraScale+ MPSoC device.

In this paper, we present a hardware-software implementation of a deep neural network for object detection based on a point cloud obtained by a LiDAR sensor. The PointPillars network was used in the research, as it is a reasonable compromise between detection accuracy and calculation complexity. The Brevitas / PyTorch tools were used for network quantisation (described in our previous paper) and the FINN tool for hardware implementation in the reprogrammable Zynq UltraScale+ MPSoC device. The obtained results show that quite a significant computation precision limitation along with a few network architecture simplifications allows the solution to be implemented on a heterogeneous embedded platform with maximum 19% AP loss in 3D, maximum 8% AP loss in BEV and execution time 375ms (the FPGA part takes 262ms). We have also compared our solution in terms of inference speed with a Vitis AI implementation proposed by Xilinx (19 Hz frame rate). Especially, we have thoroughly investigated the fundamental causes of differences in the frame rate of both solutions. The code is available at https://github.com/vision-agh/pp-finn .

In this paper we present our research on the optimisation of a deep neural network for 3D object detection in a point cloud. Techniques like quantisation and pruning available in the Brevitas and PyTorch tools were used. We performed the experiments for the PointPillars network, which offers a reasonable compromise between detection accuracy and calculation complexity. The aim of this work was to propose a variant of the network which we will ultimately implement in an FPGA device. This will allow for real-time LiDAR data processing with low energy consumption. The obtained results indicate that even a significant quantisation from 32-bit floating point to 2-bit integer in the main part of the algorithm, results in 5%-9% decrease of the detection accuracy, while allowing for almost a 16-fold reduction in size of the model.

This paper presents a GPU implementation of two foreground object segmentation algorithms: Gaussian Mixture Model (GMM) and Pixel Based Adaptive Segmenter (PBAS) modified for RGB-D data support. The simultaneous use of colour (RGB) and depth (D) data allows to improve segmentation accuracy, especially in case of colour camouflage, illumination changes and occurrence of shadows. Three GPUs were used to accelerate calculations: embedded NVIDIA Jetson TX2 (Maxwell architecture), mobile NVIDIA GeForce GTX 1050m (Pascal architecture) and efficient NVIDIA RTX 2070 (Turing architecture). Segmentation accuracy comparable to previously published works was obtained. Moreover, the use of a GPU platform allowed to get real-time image processing. In addition, the system has been adapted to work with two RGB-D sensors: RealSense D415 and D435 from Intel.

Tissue segmentation in whole-slide images is an important task in digital pathology, required for efficient and accurate computer-aided diagnostics. Precise tissue segmentation is particularly significant for a correct diagnosis in cases, when tissue structure of a specimen is very porous, such as skin specimens. In this paper, we addressed the problem of fore- and background segmentation in histopatological images of skin specimens stained with hematoxylin and eosin (H&E), which has not been solved yet, by a novel method based on a combination of statistical analysis, color thresholding, and binary morphology. We validated our algorithm on large extracts from 60 high-resolution whole slide images, with differing staining quality and captured under varying imaging conditions, from three laboratories. The size of extracts varies from 2000×1500 to 20000×30000 pixels and the number of images used in our study matches the number of H&E images used by other research teams. We compared our method to the published ones (GrabCut and FESI) and showed that our approach outperforms its counterparts (Jaccard index of 0.929 vs. 0.776 and 0.695).

In this paper we present a vision based hardware-software control system enabling autonomous landing of a mul-tirotor unmanned aerial vehicle (UAV). It allows the detection of a marked landing pad in real-time for a 1280 x 720 @ 60 fps video stream. In addition, a LiDAR sensor is used to measure the altitude above ground. A heterogeneous Zynq SoC device is used as the computing platform. The solution was tested on a number of sequences and the landing pad was detected with 96% accuracy. This research shows that a reprogrammable heterogeneous computing system is a good solution for UAVs because it enables real-time data stream processing with relatively low energy consumption.

In this research, we present a semi-supervised segmentation solution using convolutional autoencoders to solve the problem of segmentation tasks having a small number of ground-truth images. We evaluate the proposed deep network architecture for the detection of nests of nevus cells in histopathological images of skin specimens is an important step in dermatopathology. The diagnostic criteria based on the degree of uniformity and symmetry of border irregularities are particularly vital in dermatopathology, in order to distinguish between benign and malignant skin lesions. However, to the best of our knowledge, it is the first described method to segment the nests region. The novelty of our approach is not only the area of research, but, furthermore, we address a problem with a small ground-truth dataset. We propose an effective computer-vision based deep learning tool that can perform the nests segmentation based on an autoencoder architecture with two learning steps. Experimental results verified the effectiveness of the proposed approach and its ability to segment nests areas with Dice similarity coefficient 0.81, sensitivity 0.76, and specificity 0.94, which is a state-of-the-art result.

Solaris National Synchrotron Radiation Centre is a research facility that provides high quality synchrotron light. To control such a complex system it is necessary to monitor signals from various devices and subsystems. Despite the high demand for solutions to monitor the operation of centres, little work has concentrated on automatic analysis and fault detection. Anomaly detection prevents from financial loss, unplanned downtimes and in extreme cases cause damage. To address the problem a convolutional neural network (CNN) for fault detection in time series data has been proposed. The aim of the system is to identify abnormal status of sensors in certain time steps. In this study, we deploy transfer learning by examining pre-trained VGG-16, VGG-19, InceptionV3 and Xception CNN models with an adjusted densely-connected classifiers. Our database contains 336 h of signals in total which have been divided into 6300 time windows of 3 min length. The proposed solution, based on the VGG-16 architecture, detects anomalies in diagnostics signals with 92% accuracy and 85.5% precision what is a state-of-the art result. (c) 2020 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license (http:// creativecommons.org/licenses/by/4.0/).

Deep learning methods, used in machine vision challenges, often face the problem of the amount and quality of data. To address this issue, we investigate the transfer learning method. In this study, we briefly describe the idea and introduce two main strategies of transfer learning. We also present the widely-used neural network models, that in recent years performed best in ImageNet classification challenges. Furthermore, we shortly describe three different experiments from computer vision field, that confirm the developed algorithms ability to classify images with overall accuracy 87.2-95%. Achieved numbers are state-of-the-art results in melanoma thickness prediction, anomaly detection and Clostridium di cile cytotoxicity classification problems.

Over the past two decades, malignant melanoma incidence rate has dramatically risen but melanoma mortality has only recently stabilized. Due to its propensity to metastasize and lack of effective therapies for most patients with advanced disease, early detection of melanoma is a clinical imperative. Thickness is one of the most important factor in melanoma prognosis and it is used to establish the size of the surgical margin, as well as to select patients for sentinel lymph node biopsy. However, little work has concentrated on the evaluation of melanoma thickness both from the clinical as well as computer-aided diagnostic side. To address this problem, we propose an effective computer-vision based machine learning tool that can perform the preoperative evaluation of melanoma thickness. The novelty of our approach is that we directly predict the thickness of the skin lesion into one of three classes: less than 0.75 mm, 0.76-1.5 mm, and greater that 1.5 mm. In this study, we use transfer learning of the pre-trained, adapted to our application VGG-19 convolutional neural network (CNN) with an adjusted densely-connected classifier. Due to the limited data we investigate the transfer learning method where we apply knowledge from model trained on a different task. Our database contains 244 dermoscopy images. Experiments confirm the developed algorithms ability to classify skin lesion thickness with 87.2% overall accuracy what is a state-of-the-art result in melanoma thickness prediction.

In this paper the design of an embedded vision system for structure from motion (SFM) computation is presented. The solution allows for scene depth reconstruction based on two consecutive video frames registered by a moving camera. The module is developed for a Xilinx Zynq SoC (System on Chip) device and is divided into a hardware and software part. The programmable logic (PL) is used mainly for detect and match features. Fundamental matrix estimation and triangulation are executed in the processing system (PS-ARM based). The module is dedicated to navigation of unmanned aerial vehicles (UAV)-autonomous flight and landing. The system is adapted to process a 1280×720 pixel video stream @ 60 frames per second in realtime.

The article demonstrates the usefulness of heterogeneous System on Chip (SoC) devices in smart cameras used in intelligent transportation systems (ITS). In a compact, energy efficient system the following exemplary algorithms were implemented: vehicle queue length estimation, vehicle detection, vehicle counting and speed estimation (using multiple virtual detection lines), as well as vehicle type (local binary features and SVM classifier) and colour (k-means classifier and YCbCr colourspace analysis) recognition. The solution exploits the hardware–software architecture, i.e. the combination of reconfigurable resources and the efficient ARM processor. Most of the modules were implemented in hardware, using Verilog HDL, taking full advantage of the possible parallelization and pipeline, which allowed to obtain real-time image processing. The ARM processor is responsible for executing some parts of the algorithm, i.e. high-level image processing and analysis, as well as for communication with the external systems (e.g. traffic lights controllers). The demonstrated results indicate that modern SoC systems are a very interesting platform for advanced ITS systems and other advanced embedded image processing, analysis and recognition applications.

High Efficiency Video Coding (HEVC), a modern video compression standard, exceeds the predecessor H. 264 in efficiency by 50%, but with cost of increased complexity. It is one of main research topics for FPGA engineers working on image compression algorithms. On the other hand high-level synthesis tools after few years of lower interest from the industry and academic research, started to gain more of it recently. This paper presents FPGA implementation of HEVC 2D Inverse DCT transform implemented on Xilinx Virtex-6 using Impulse C high level language. Achieved results exceed 1080p@30fps with relatively high FPGA clock frequency and moderate resource usage.

In this paper a hardware-software design of an object tracking system, which uses a moving camera is presented. The solution is implemented on the Zybo development board with the Zynq SoC (System on Chip) device from Xilinx. The object's position is used to control two servomotors, which constitute a pan-tilt mounting of the camera. The proposed system is able to process a 1280 × 720 @ 60 fps video stream in real-time and track moving objects.

In this paper a hardware-software abandoned object detection vision system implemented in the Zynq SoC (System on Chip) device is presented. First, the solution was implemented in C++ and run as a bare metal application on the ARM processor core of the Zynq (using floating and fixed-point computations). For the target video stream 1280 χ 720 @ 50 fps (74.25 MHz pixel clock) it reached only 2 fps. Therefore, to speed-up the application, it was decided to move some of the image processing and analysis operations to the programmable logic. This allowed to obtain real-time image processing i.e. 50 fps, with power consumption of less than 4W.