Heads-up displays that are ‘see-through’ and ‘curved’ and capable of displaying 3D contents are considered crucial for augmented reality-based navigation in automobiles. Here we report the development, calibration and experimental evaluation of a 3D display system that satisfies the above requirements. Integral imaging is used as the 3D display technique, which is realized using a flexible ‘concave-micro-mirror array’ screen (equivalent of a ‘micro-lens array’, but working in reflection mode). The screen itself is fabricated as a holographic optical element. The holographic nature of the screen enables a ‘see-through’ effect. The 3D content to be displayed is served by a 2D projector as integral images. A novel calibration method is developed which employs diffusive markers, that are invisible to the naked eye, being placed at one corner of each elemental micro-mirror. The calibration enables proper treatment of the effects and artifacts caused by screen ‘curvature’, but the presence of markers itself does not degrade the display characteristics. A curved micro-mirror array screen of size 10 cm × 10 cm consisting of 100 × 200 elemental concave mirrors is fabricated as a flexible holographic optical element with diffusive markers of size 300 m × 300 m. The screen, when illuminated with a projector (that serves integral images), was able to reconstruct a 3D scene of size 10 cm × 10 cm with a depth of 5 cm. The novel calibration method employing diffusive markers demonstrates significant improvement in calibration accuracy. The curved and see-through nature of the display screen makes it a good choice for windshield displays. The reported system requires further improvements in enlarging the screen size and increasing depth of the 3D scene in order to meet real-world requirements, which can be achieved by scaling-up the system.

Capturing an event from multiple camera angles can give a viewer the most complete and interesting picture of that event. To be suitable for broadcasting, a human director needs to decide what to show at each point in time. This can become cumbersome with an increasing number of camera angles. The introduction of omnidirectional or wide-angle cameras has allowed for events to be captured more completely, making it even more difficult for the director to pick a good shot. In this paper, a system is presented that, given multiple ultra-high resolution video streams of an event, can generate a visually pleasing sequence of shots that manages to follow the relevant action of an event. Due to the algorithm being general purpose, it can be applied to most scenarios that feature humans. The proposed method allows for online processing when real-time broadcasting is required, as well as offline processing when the quality of the camera operation is the priority. Object detection is used to detect humans and other objects of interest in the input streams. Detected persons of interest, along with a set of rules based on cinematic conventions, are used to determine which video stream to show and what part of that stream is virtually framed. The user can provide a number of settings that determine how these rules are interpreted. The system is able to handle input from different wide-angle video streams by removing lens distortions. Using a user study it is shown, for a number of different scenarios, that the proposed automated director is able to capture an event with aesthetically pleasing video compositions and human-like shot switching behavior.

Light field 3D displays require a precise alignment between the display source and the micromirror-array screen for error free 3D visualization. Hence, calibrating the system using an external camera becomes necessary, before displaying any 3D contents. The inter-dependency of the intrinsic and extrinsic parameters of display-source, calibration-camera, and micromirror-array screen, makes the calibration process very complex and error-prone. Thus, several assumptions are made with regard to the display setup, in order to simplify the calibration. A fully automatic calibration method based on several such assumptions was reported by us earlier. Here, in this paper, we report a method that uses no such assumptions, but yields a better calibration. The proposed method adapts an optical solution where the micromirror-array screen is fabricated as a computer generated hologram with a tiny diffuser engraved at one corner of each elemental micromirror in the array. The calibration algorithm uses these diffusing areas as markers to determine the relation between the pixels of display source and the mirrors in the micromirror-array screen. Calibration results show that virtually reconstructed 3D scenes align well with the real world contents, and are free from any distortion. This method also eliminates the position dependency of display source, calibration-camera, and mirror-array screen during calibration, which enables easy setup of the display system.

Recent advances in the creation of microlens arrays as holographic optical elements allow the creation of projector-based see-through light field displays suitable for augmented reality. These systems require an accurate calibration of the projector with relation to the microlens array, as any small misalignment causes the 3D reconstruction to fail. The methods reported so far require precise placement of the calibration camera w.r.t. the lens array screen, which affects the display configuration. We propose a calibration approach which is more robust, and which allows free camera placement. Hence, it does not limit the capabilities of the system. Both a homography-based technique and structured light play a central role in realizing such a method. The method was tested on a projection-based integral imaging display system consisting of a consumer-grade projector and a digitally designed holographic optical element based micromirror array screen. The calibration method compensates for the lens distortion, intrinsics, and positioning of the projector with relation to the screen. The method uses a single camera and does not require the use of obtrusive markers as reference. We give an in-depth explanation of the different steps of the algorithm, and verify the calibration using both a simulated and a real-world setup.

See-through head-up 3D display is being reported, which uses a commercial projector as the light source and a lens array in the form of holographic optical element (HOE). By introducing novel technique in the fabrication process of the HOE based lens array, we have significantly simplified the display system by not using any collimation optics. Light field reconstructions of a 3D scene of size 20cm (Width) × 10cm (Height) × 5 cm (Depth) is successfully demonstrated with a view angle of 10-degrees.

A hologram of a scene can be digitally created by using a large set of images of that scene. Since capturing such a large amount is infeasible to accomplish, one may use view synthesis approaches to reduce the number of cameras and generate the missing views. We propose a view interpolation algorithm that creates views inside the scene, based on a sparse set of camera images. This allows the objects to pop out of the holographic display. We show that our approach outperforms existing view synthesis approaches and show the applicability on holographic stereograms.

We introduce an effective technique to enhance the images captured underwater and degraded due to the medium scattering and absorption. Our method is a single image approach that does not require specialized hardware or knowledge about the underwater conditions or scene structure. It builds on the blending of two images that are directly derived from a color-compensated and white-balanced version of the original degraded image. The two images to fusion, as well as their associated weight maps, are defined to promote the transfer of edges and color contrast to the output image. To avoid that the sharp weight map transitions create artifacts in the low frequency components of the reconstructed image, we also adapt a multiscale fusion strategy. Our extensive qualitative and quantitative evaluation reveals that our enhanced images and videos are characterized by better exposedness of the dark regions, improved global contrast, and edges sharpness. Our validation also proves that our algorithm is reasonably independent of the camera settings, and improves the accuracy of several image processing applications, such as image segmentation and keypoint matching.

Concave micro-mirror arrays fabricated as holographic optical elements are used in projector-based light field displays due to their see-through characteristics. The optical axes of each micro-mirror in the array are usually made parallel to each other, which simplifies the fabrication, integral image rendering, and calibration process. However, this demands that the beam from the projector be collimated and made parallel to the optical axis of each elemental micro-mirror. This requires additional collimation optics, which puts serious limitations on the size of the display. In this Letter, we propose a solution to the above issue by introducing a new method to fabricate holographic concave micro-mirror array sheets and explain how they work in detail. 3D light field reconstructions of the size 20  cm×10  cm and 6 cm in depth are achieved using a conventional projector without any collimation optics.

This paper describes the concepts and results implemented by the European FP7 Dreamspace project. Dreamspace develops a new platform and tools for collaborative virtual production of visual effects in film and TV and new immersive experiences. The aim of the project is to enable creative professionals to combine live performances, video, and computer-generated imagery in realtime. In particular, t...

A calibration technique for integral photography(IP) based 3D display that uses a commercial projector and digitally designed holographic optical element (DDHOE) as lens array is reported. This is an automated method that compensates for both lens distortion and DDHOE-projector alignment errors which shows huge improvement in reconstruction quality. This method requires less number of input images for calibration compared to existing ones.

In recent years there is a growing interest in the generation of virtual views from a limited set of input cameras. This is especially useful for applications such as Free Viewpoint Navigation and light field displays [Tanimoto 2015]. The latter often requires tens to hundreds of input views, while it is often not feasible to record with as many cameras. View interpolation algorithms often traverse a set of depths to find correspondences between the input images [Stankiewicz et al. 2013; Goorts et al. 2013]. Most algorithms choose a uniform set of depths to traverse (as shown in Figure 2(a)), but this often leads to an excessive amount of unnecessary calculations in regions where no objects are located. It also results in an increased amount of mismatches, and thus, inaccuracies in the generated views. These problems also occur when a too large depth range is selected. Hence, typically a depth range that encloses the scene tightly is manually selected to mitigate these errors. A depth distribution that organizes the depth layers around the objects in the scene, as shown in Figure 2(b), would reduce these errors and decrease the number of computations by reducing the number of depths to search through. [Goorts et al. 2013] determine a nonuniform global depth distribution by reusing the generated depth information from the previous time stamp. This makes the algorithm dependent on previous results.

Real-time large scale continuous image and geometry based data visualization, with an uninterrupted content delivery, quality and rendering, on home and mobile devices is difficult or even mostly impossible because of the low processing capabilities of these hardware devices. However, a gracefully simplified version of the same data can enable us to view the content without significant quality degradation. To do this in a graceful manner, we extended a well-known concept - called ’billboard cloud’ - for animated scene data and implemented this technique using the capabilities of the GPU to generate the simplified versions of large scale data sets.