Multispectral imaging technology analyzes for each pixel a wide spectrum of light and provides more spectral information compared to traditional RGB images. Most current Unmanned Aerial Vehicles (UAV) camera systems are limited by the number of spectral bands (≤10 bands) and are usually not fully integrated with the ground controller to provide a live view of the spectral data. We have developed a compact multispectral camera system which has two CMV2K 4x4 snapshot mosaic sensors internally, providing 31 bands in total covering the visible and near-infrared spectral range (460-860nm). It is compatible with (but not limited to) the DJI M600 and can be easily mounted to the drone. Our system is fully integrated with the drone, providing stable and consistent communication between the flight controller, the drone/UAV, and our camera payload. With our camera control application on an Android tablet connected to the flight controller, users can easily control the camera system with a live view of the data and many useful information including histogram, sensor temperature, etc. The system acquires images at a maximum framerate of 2x20 fps and saves them on an internal storage of 1T Byte. The GPS data from the drone is logged with our system automatically. After the flight, data can be easily transferred to an external hard disk. Then the data can be visualized and processed using our software into single multispectral cubes and one stitched multispectral cube with a data quality report and a stitching report.

This paper presents the impact of a multi-exposure method for a hyperspectral imaging camera in the visual-near infrared range. Multi-exposure helps deal with the dynamic range requirements coming from uneven scene illumination, imbalanced light source energy distribution per band and diverse object reflectivity levels. We show the benefits of applying multi-exposure for different light sources (halogen and LED) and varied scene types at the application side. Our results show that improved SNR and decreased spectral variability are achieved. This results in increased discrimination capacity at application level, with increases of up to 10% in the mean classification accuracy.

SWIR (Short Wave Infrared) imaging can be of great use in precision agriculture, food processing and recycling industry, among other fields. However, hyperspectral SWIR cameras are costly and bulky, preventing their widespread deployment on the field. To answer the market need for compact and cost-efficient hyperspectral cameras covering the SWIR range, imec and SCD have joined efforts to develop a novel integration approach combining imec know-how in pixel level patterned thin film spectral filter technology, with SCD's InGaAs technology.The here presented line-scan SWIR hyperspectral camera covers the 1.1-1.65 mu m range with 100+ bands and a spectral resolution better than 10 nm. This imager uses a set of patterned Fabry-Perot interferometers processed using semiconductor grade thin-film technology. The optical filters are then integrated directly on top of the sensing side of the InGaAs detector with high accuracy and with a minimum gap between filters and Focal Plane Array to limit cross-talk. The resulting line-scan camera, measuring only 70x62x60 mm and with a weight below 0.5kg, is the lightest and most compact SWIR hyperspectral camera on the market. Full sensor readout can be performed at up to 350 fps. An imecpatented SnapScan system with internal scanning was also developed, capable of acquiring data cubes of 640x512x128 pixels in a second. Maximum cube size is 1200x640x128. By selecting a subset of contiguous spectral bands and a reduced spatial resolution the sensor could be operated @ +1000 fps, for example enabling cube acquisitions of 320x512x64 in less than 300 ms.

Imec has developed compact hyperspectral image sensors where optical filters are monolithically integrated over standard off-the-shelf CMOS image sensors. These filters are implemented at individual pixel level and are Fabry-Perot interferometers. Due to the monolithic integration of optical filters on the image sensors, the characterization procedures and camera response model developed for traditional sensors cannot be directly used for these hyperspectral sensors. In this paper, we present the procedures used for characterizing these sensors and deriving a suitable camera response model for them.

Imec has developed a process for the monolithic integration of optical filters on top of CMOS image sensors, leading to compact, cost-efficient and faster hyperspectral cameras. Different prototype sensors are available, most notably a 600-1000 nm line-scan imager, and two mosaic sensors: a 4x4 VIS (470-620 nm range) and a 5x5 VNIR (600-1000 nm). In response to the users' demand for a single sensor able to cover both the VIS and NIR ranges, further developments have been made to enable more demanding applications. As a result, this paper presents the latest addition to imec's family of monolithically-integrated hyperspectral sensors: a line scan sensor covering the range 470-900 nm.This new prototype sensor can acquire hyperspectral image cubes of 2048 pixels over 192 bands (128 bands for the 600-900 nm range, and 64 bands for the 470-620 nm range) at 340 cubes per second for normal machine vision illumination levels.

To enable industrial adoption of hyperspectral imaging we have developed a unique integrated hyperspectral filter/imager technology. The spectral filters are monolithically deposited/integrated on top of CMOS imagers at wafer level. The materials of the filters are chosen such that they are compatible with the production flows available in most CMOS foundries. The result is a compact & fast hyperspectral imager made with low-cost CMOS process technology. We have demonstrated this hyperspectral technology on two specific instances – a wedge based linescan hyperspectral imager and a tile based snap-shot imager. The line-scan imager is based on a CMOSIS CMV 4000 image sensor and the spectral specifications are 100 spectral bands in the range of 600-1000nm and the FWHM of each band is around 10nm. The snapshot imager is based on a CMOSIS CMV 2000 image sensor with the following spectral specifications: 32 spectral bands in the range of 600-1000nm and also with FWHM of each band around 10nm. Furthermore, both technology is flexible such that many system parameters like number of spectral bands, layout of the filters, FWHM of the filters and the spectral range can be tuned to match specific application requirements. 1. Integrated Spectral Filter Approach Hyperspectral imaging is an advanced imaging technique which captures and processes multiple narrow band images over a spectral range. Capturing spectral images of an object enables detailed analysis and identification of the objects as often different objects contain unique information at different wavelengths. The potential of hyperspectral imaging has been demonstrated for several applications using laboratory setups, it is currently mostly still a scientific tool. Indeed, many of the commercial hyperspectral cameras available today are targeted for the research market, e.g. remote sensing [1] and food science [2]. The adoption of hyperspectral imaging in mainstream industrial applications like machine vision or biomedical has so far been limited, due to the lack of fast, compact, and cost-effective hyperspectral cameras with adequate specifications. (a) (b) Spectral range 600-1000nm # Spectral bands 100 FWHM < 10nm, using collimated light Filter transmission efficiency ~ 85% Imager CMOSIS CMOS CMV 4000 imager #lines/ spectral band 16 #spatial pixels/line 2048 scan rate in #lines/sec 2880

In this paper we present a technique to accurately build a 3D hyperspectral image cube from a 2D imager overlaid with a wedge filter with up to hundreds of spectral bands, providing time-multiplexed data through scanning. The correctness of the spectral curve of each pixel in the physical scene, being the combination of its spectral information captured over different time stamps, is directly related to the alignment accuracy and scanning sensitivity. To overcome the accumulated alignment errors from scanning inaccuracies, frequency- dependent scaling from lens, spectral band separations and the imager’s spectral filter technology limitations, we have designed a new image alignment algorithm based on Random Sample Consensus (RANSAC) model fitting. It estimates many mechanical and optical system model parameters with image feature matching over the spectral bands, ensuring high immunity against the spectral reflectance variations, noise, motion-blur, blur etc. The estimated system model parameters are used to align the images captured over different bands in the 3D hypercube, reducing the average alignment error to 0.5 pixels, much below the alignment error obtained with state-of-the-art techniques. The image feature correspondences between the images in different bands of the same object are consistently produced, resulting in a hardware-software co-designed hyperspectral imager system, conciliating high quality and correct spectral curve responses with low-cost.

Colony counting is a procedure used in microbiology laboratories for food quality monitoring, environmental management, etc. Its purpose is to detect the level of contamination due to the presence and growth of bacteria, yeasts and molds in a given product. Current automated counters require a tedious training and setup procedure per product and bacteria type and do not cope well with diversity. This contrasts with the setting at microbiology laboratories, where a wide variety of food and bacteria types have to be screened on a daily basis. To overcome the limitations of current systems, we propose the use of hyperspectral imaging technology and examine the spectral variations induced by factors such as illumination, bacteria type, food source and age and type of the agar. To this end, we perform experiments making use of two alternative hyperspectral processing pipelines and compare our classification results to those yielded by color imagery. Our results show that colony counting may be automated through the automatic recovery of the illuminant power spectrum and reflectance. This is consistent with the notion that the recovery of the illuminant should minimize the variations in the spectra due to reflections, shadows and other photometric artifacts. We also illustrate how, with the reflectance at hand, the colonies can be counted making use of classical segmentation and classification algorithms.

User satisfaction is a key factor in the success of novel multimedia services. Yet, to enable service providers and network operators to control and maximize the quality (QoS, QoE) of delivered video streams, quite some challenges remain. In this paper, we particularly focus on three of them. First of all, objectively measuring video quality requires appropriate quality metrics and methods of assessing them in a real-time fashion. Secondly, the recent Scalable Video Coding (SVC) format opens opportunities for adapting video to the available (network) resources, yet the appropriate configuration of video encoding as well as real-time streaming adaptation are largely unaddressed research areas. Thirdly, while bandwidth reservation mechanisms in access/core networks do exist, service providers lack a means for guaranteeing QoS in the increasingly complex home networks (which they are not in full control of). In this paper we offer a broad view on these interrelated issues, by presenting the developments originating in a Flemish research project (including proof-of-concept demonstrations). From a developmental perspective, we propose an architecture combining a real-time video quality monitoring platform, on-the-fly adaptation (optimizing the video quality) and QoS reservation in a heterogeneous home network based on UPnP QoS v3. From a research perspective, we propose a new subjective test procedure that revealed user preference for temporal scalability over quality scalability. In addition, an extensive study on optimizing HD SVC encoding in IPTV scenarios with fluctuating bandwidth showed that under certain bandwidth constraints (prohibiting sufficient fidelity) spatial scalability is a better option than quality scalability.

A major limitation for wireless video communication on portable devices is the limited energy budget. For this reason, efficient usage of the scarce energy becomes a critical design constraint, in addition to meeting the Quality of Service constraints related to the video quality. In this chapter the authors focus on minimizing the energy cost of the two main energy consumers in the handheld wireless video device: the video encoding and wireless communication tasks. For this purpose, they present a cross-layer approach that explores the tradeoff between coding and communication energies. They then exploit the Rate-Distortion-Complexity tradeoffs and flexibility of the Scalable Video Codec. The results show that by adapting the codec configuration at runtime to the specific scenarios up to 50% of the total energy can be saved with marginal video quality loss. Moreover, the approach presented is of low complexity and easily deployable in practical systems.

A major limitation for wireless video communication over portable devices is the limited energy supply. For this reason, an efficient energy usage becomes a critical issue. In this paper we focus on the energy minimization of the two main energy consumers at the device: video encoding and wireless communication tasks. For this purpose, we develop a cross-layer approach that explores the tradeoff between coding and communication energies. We then exploit the Power-Rate tradeoffs and flexibility of the Scalable Video Codec. Our results show that by adapting the codec configuration at runtime to the specific scenarios we can save up to 40% of the total energy without video quality loss. Moreover, our approach is of low complexity and easily deployable.

The Scalable Video Coding extends the H.264/AVC video coding standard by providing temporal, spatial and quality scalability. A set of new tools, such as key picture and inter-layer prediction are introduced in SVC to improve either rate-distortion performance or error resilience. This paper evaluates the impact of these tools on the performance of an optimized SVC decoder, in terms of rate, distortion, and computational complexity. The results facilitate the decision making process of choosing suitable SVC configurations under different application scenarios.

The consultative committee for space data systems (CCSDS) data compression working group has recently adopted a recommendation for image data compression, with a final release expected in 2005. The algorithm adopted in the recommendation consists of a two-dimensional discrete wavelet transform of the image, followed by progressive bit-plane coding of the transformed data. The algorithm can provide both lossless and lossy compression, and allows a user to directly control the compressed data volume or the fidelity with which the wavelet-transformed data can be reconstructed. The algorithm is suitable for both frame-based image data and scan-based sensor data, and has applications for near-Earth and deep-space missions. The standard will be accompanied by free software sources on a future Web site. An application-specific integrated circuit (ASIC) implementation of the compressor is currently under development. This paper describes the compression algorithm along with the requirements that drove the selection of the algorithm. Performance results and comparisons with other compressors are given for a test set of space images.

H.264/AVC video coding standard promises improved coding efficiency compared with other standards such as MPEG-2. However, its computational complexity is also increased significantly. Efficiently mapping H.264/AVC decoder onto a flexible platform presents a big challenge to existing architectures and design methodology. This paper describes the process and results of mapping H.264/AVC decoder onto the ADRES architecture (Mei et al., 2003), which is a flexible coarse-grained reconfigurable architecture template that tightly couples a VLIW processor and a coarse-grained array.

The realization of new MPEG-4 functionality, applicable to three-dimensional graphics texture compression and image database access over the Internet, is demonstrated on a heterogeneous platform with several unique features. First, applying our system-level design methodologies effectively removes all data transfer and storage overhead that comprises the main bottleneck in the original system description. Second, a first-of-a-kind application specific solution, called Ozone, accelerates the embedded-zero-tree based encoding and is capable of compressing 30 color CIF images per second. The entire application is running on the Ozone coupled to a PC.

The FlexWave-II has been developed as a dedicated image compression component for spaceborn applications, enabling a multitude of application scenarios, including lossless and lossy compression. The FlexWave-II provides scalable compression, allowing gradual enhancement or de gradation of the image quality in a programmable way. A wavelet-based compression scheme has been selected because of the intrinsic scalable characteristics. Moreover the compression criteria can be tuned separately for optimal measurement and visual data compression.The FlexWave-II provides full scalability features and high processing performance. It supports push-broom image processing. The wavelet transform engine is capable of computing up to 5 levels of wavelet transform with 5/3-, 9/3- or 9/7-tap wavelet filters, for image sizes as large as 1k x 1k pixels. On an FPGA implementation, clocked on 41 MHz, a processing performance of up to 10 Mpixels/second was measured. The wavelet compression engine allows two compression modes: a fixed compression ratio mode optimised for user-defined criteria and a fixed quantisation mode with user defined quantisation tables.