Inspired by the recent work in language modeling, we investigate the effects of a set of regularization techniques on the performance of a recurrent neural network based image captioning model. Using these techniques, we achieve 13 Bleu-4 points improvements over using no regularizations. We show that our model does not suffer from loss-evaluation mismatch and also connect the model performance to dataset properties by running experiments on MSCOCO dataset. Further, we propose a human in the loop image captioning system as an alternative way to improve the model performance. Using only the first two tokens of a reference sentence of an image, we improve Bleu-4 score of our best model by 57 points with this hybrid system.

This study aims to provide an overview on the intersection and interaction between architecture, urban modeling, planning fields and computer vision field. The reflection of the methods and approaches of fields such as visual recognition, natural language processing, data mining and data visualization onto architecture and urban studies are investigated and potentials of inter/transdisciplinary encounters are unfolded through research questions. This study can be considered as an initial attempt to constitute a common and sustainable platform for advancing efficient inter/transdisciplinary and cross-domain collaborations.

Good representative dictionaries is the most critical part of the BoVW: Bag of Visual Words scheme, used for such tasks as category identification. The paradigm of learning dictionaries from datasets is by far the most widely used approach and there exists a plethora of methods to this effect. Dictionary learning methods demand abundant data, and when the amount of training data is limited, the quality of dictionaries and consequently the performance of BoVW methods suffer. A much less explored path for creating visual dictionaries starts from the knowledge of primitives in appearance models and creates families of parametric shape models. In this work, we develop shape models starting from a small number of primitives and develop a visual dictionary using various nonlinear operations and nonlinear combinations. Compared with the existing model-driven schemes, our method is able to represent and characterize images in various image understanding applications with competitive, and often better performance.

We designed SymPaD framework, a model-driven visual dictionary construction and description method, with new shape models and quantized shape library. We demonstrate that, with this new design, the most current model-driven dictionary construction method is outperformed with even smaller dictionary in object recognition and image retrieval tasks.

Generalized Hough transform, when applied to object detection, recognition and pose estimation, can be susceptible to spurious voting depending on the Hough space to be used and hypotheses to be voted. This often necessitates additional computational steps like non-maxima suppression and geometric consistency checks, which can be costly and prevent voting based methods from being precise and scalable for large numbers of target classes and crowded scenes. In this paper, we propose an efficient and refined Hough transform for simultaneous detection, recognition and exact pose estimation, which can efficiently accommodate up to multiple tens of co-visible query instances and multiple thousands of visually similar classes. Specifically, we match SURF features from a given query image to a database of model features with known poses, and in contrast to existing techniques, for each matched pair, we analytically compute a concise set of 6 degrees-of-freedom pose hypotheses, for which the geometric relationship of the correspondence remains invariant. We also introduce an indirect but equivalent representation for those correspondence-specific poses, termed as feature aligning affine transformations, which results in a Hough voting scheme as cheap and refined as line drawing in raster grids. Owing to minimized voting redundancy, we can obtain a very sparse and stable Hough image, which can be readily used to read off instances and poses without dedicated steps of non-maxima suppression and geometric verification. Experimented on an extensive Grocery Products dataset, our method significantly outperforms the sate-of-the-art with near real time overall cost.

We propose a new local image descriptor named SymPaD for image understanding. SymPaD is a probability vector associated with a given image pixel and represents the attachment of the pixel to a previously designed shape repertoire. As such the approach is model-driven. The SymPad descriptor is illumination and rotation invariant, and extremely flexible on extending the repertoire with any parametrically generated geometrical shapes and any desired additional transformation types.

Emerging cost-efficient depth sensor technologies reveal new possibilities to cope with difficulties in action recognition. Depth information improves the quality of skeleton detec- tion process, hence, pose estimation can be done more efficiently. Recently many studies fo- cus on temporal analyses over estimated skeleton poses to recognize actions. In this paper we have an inclusive study of the spatiotemporal kinematic features and propose an action recognition framework with feature selection capability to deal with the multitudinous of features by leveraging data mining capabilities of random decision forests. We describe human motion via a rich collection of kinematic feature time-series computed from the skel- etal representation of the body in motion. We discriminatively optimize a random decision forest model over this collection to identify the most effective subset of features, localized both in time and space. Later, we train a support vector machine classifier on the selected features. This approach improves upon the baseline performance obtained using the whole feature set with a significantly less number of features (one tenth of the original). To justify our method we test the framework on various datasets and compared it with state-of-the- art. On MSRC-12 dataset (25) (12 classes), our method achieves 94% accuracy. On the WorkoutSU-10 dataset (28), collected by our group (10 physical exercise classes), the ac- curacy is 98%. On MSR Action3D dataset (9) (20 classes) we obtain 87% average accura- cy and for UTKinect-Action dataset (10) (10 classes) the accuracy is 92%. Other than regular activities, we also tried our approach to detect a falling person using the dataset which we recorded as an extension to our original dataset. We test how our method adjusts on different types of actions and we obtained promising results for this type of action. We discuss that our approach provides insights on the spatiotemporal dynamics of human ac- tions and can be used to as part of different applications especially for rehabilitation of pa- tients.

In this study we propose a model-driven codebook generation method used to assign probability scores to pixels in order to represent underlying local shapes they reside in. In the first version of the symbol library we limited ourselves to photometric and similarity transformations applied on eight prototypical shapes of flat plateau, ramp, valley, ridge, circular and elliptic respectively pit and hill and used randomized decision forest as the statistical classifier to compute shape class ambiguity of each pixel. We achieved similar to 90% accuracy in identification of known objects from alternate views. For recognition of unknown objects, we could not outperform texture, global and local shape methods, but only color-based method in average, however we could outperform existing methods for three individual categories by exploiting multispectral analysis and multiscale approach. We present a progress plan to be accomplished as a future work to improve the proposed approach further.

In this work, we primarily aim at estimating the performance of SVM-based gender recognition using widely used DCT and LBP facial features, as faithful as possible. The SVM classifier has been trained and cross-validated on the FERET database containing 2720 instances, while for testing, the LFW database containing over 13000 instances has been used. We have observed that the over 95% cross-validation performance on FERET is overly optimistic as compared to the true test performance of %78 on LFW. Additionally, we have used random decision forests as a discriminative feature selection scheme and we have shown that similar performance can be maintained while reducing the original number of features significantly. As a by-product, the scheme can also be used to localize the most discriminative facial gender features.

Humans carry out object recognition as a very primitive task. However, knowing all the detailed information regarding an object is mostly not possible. Therefore, object recognition and linking the object to associated data is a popular research area. In this study, we have measured leaf recognition performances of different feature and data sets using image processing techniques. We have studied the effects of general shape features, Fourier descriptors and MDM-A feature sets for the solution of this problem. We have also measured performance evaluation by fusing these features. Eventually, we have observed a compatible performance with similar studies.

In this paper, we present an action recognition framework leveraging data mining capabilities of random decision forests trained on kinematic features. We describe human motion via a rich collection of kinematic feature time-series computed from the skeletal representation of the body in motion. We discriminatively optimize a random decision forest model over this collection to identify the most effective subset of features, localized both in time and space. Later, we train a support vector machine classifier on the selected features. This approach improves upon the baseline performance obtained using the whole feature set with a significantly less number of features (one tenth of the original). On MSRC-12 dataset (12 classes), our method achieves 94% accuracy. On the WorkoutSU-10 dataset, collected by our group, the accuracy is 98%. The approach can also be used to provide insights on the spatiotemporal dynamics of human actions.

In this paper, we develop a graph-based method to align two dynamic skeleton sequences, and apply it to both action recognition tasks as well as to the objective quantification of the goodness of the action performance. The automated measurement of "action quality" has potential to be used to monitor action imitations, for example, during a physical therapy. We seek matches between a query sequence and model sequences selected with graph mining. The best matches are obtained through minimizing an energy function that jointly measures space and time domain deformations. This measure has been used for recognizing actions, for separating acceptable and unacceptable action performances, or as a continuous quantification of the action performance goodness. Experimental evaluation demonstrates the improved results of our scheme vis-à-vis its nearest competitors. Furthermore, a plausible relationship has been obtained between action perturbation, given by the joint noise variances, and quality measure, given by matching energies averaged over a sequence.

Clinical experience sharing (CES) is a useful concept for both medical treatment and medical education purposes. One way of implementing CES is through the use of content based case retrieval (CBCR), where database of medical cases is browsed for case instances that are similar to the input query case. In this study, we introduce a new project called case retrieval in radiology (CaReRa), which aims at implementing CES for liver cases. We particularly focus on 3D liver images acquired by computed tomography (CT) and lay the foundations of a conceptual system outputting a ranked list of results for a given query case, formulated in this work as a liver lesion. A list of CT image features serves as computer generated descriptors together with user expressed annotations collected using a novel ontology of liver for radiology (ONLIRA). A two stage approach is proposed to utilize these two types of descriptors in cascade, namely semantic framing and similarity ranking. Initial retrieval performance results confirm the importance of ontology based descriptors, while also highlights the foci of future work needed to overcome the weaknesses.

In this study the focus was on the one of the latest trending topics, 3D object recognition, which became trending by the developments in the 3d imaging technologies. One of the methods that is used was developed directly for 3D object recognition and the other one was developed for 2D leaf recognition. The second one was adapted for 3D object recognition. Both methods are global methods. Their separate and fused performances were examined. On full object models both methods performs well, and due to their structure they are promising methods for partial object models.

In this work, the curve compression problem is approached with a model-based probabilistic framework. We propose three different models. The proposed models can be used for purposes such as feature extraction or compression. The first model we propose is basically a Bayesian regression model for fitting piece-wise defined segments. The second model unifies clustering with regression. The third model combines Hidden Markov Models with regression via adding temporal connectivity to the second model. Since these models unify the mentioned paradigms, we believe that this work may be interesting from the Bayesian modeling perspective, besides the usefulness of the proposed models for curve compression applications.

One of the greatest risk for seniors is to fall when they are alone. Detection of the fall event and early intervention minimizes the effect of the fall. The fall event could be observed without necessity of a companion thanks to advances in computer vision. This paper aims to detect the fall events using information received from motion history and shape change. Additionally, the effect of multi-camera network to this approach is analyzed.

In this work, we present a hands clapping rhythm analysis module of a video analytics framework, which monitors elderly patients and automatically collect statistical data about patient activities. Hands clapping activity is analyzed in terms of frequency of clapping, extent of clapping, and direction change. A severe level Alzheimer patient was chosen from an elderly house. The main idea makes use of optical flow vectors which represent the motion change of image features in consecutive frames. The algorithm steps are composed of detecting optical flow vectors in skin regions, clustering based on the direction, calculating the average flow vector in each cluster and observing these vectors over time. The magnitude of the average flow represents the speed of motion. In the supplementary figure, handsclapping.png, the experimental results are presented. Hands motion of the patient on the right has been observed for 100 frames (4 secs). Input hands region, detected optical flows are demonstrated, followed by the two resultant motion flow groups depicted by black and white regions. The patient is active during 100 frames and claps hands eight times, two of which are long extent clapping, when the patient is very happy. In the graphs, blue lines represent the motion of right hand, while red lines represent left hand. The occurrence of clapping hands is detected by finding the instant, when right hand moves in (+) direction and changes direction to (-); and left hand moves in (-) direction and changes to (+); and the speed of each hand is greater than 2 units. It happens at frames: 4, 11,17,28,53,63,68,87. The graph in the bottom shows the distance traveled by each hand per frame. The symmetry in motion waves of right and left hand depicts the clapping motion characteristics and validates effectiveness of the proposed method.

Diagnostic radiology requires accurate interpretation of complex signals in medical images. Content-based image retrieval (CBIR) techniques could be valuable to radiologists in assessing medical images by identifying similar images in large archives that could assist with decision support. Many advances have occurred in CBIR, and a variety of systems have appeared in nonmedical domains; however, permeation of these methods into radiology has been limited. Our goal in this review is to survey CBIR methods and systems from the perspective of application to radiology and to identify approaches developed in nonmedical applications that could be translated to radiology. Radiology images pose specific challenges compared with images in the consumer domain; they contain varied, rich, and often subtle features that need to be recognized in assessing image similarity. Radiology images also provide rich opportunities for CBIR: rich metadata about image semantics are provided by radiologists, and this information is not yet being used to its fullest advantage in CBIR systems. By integrating pixel-based and metadata-based image feature analysis, substantial advances of CBIR in medicine could ensue, with CBIR systems becoming an important tool in radiology practice.

In this work, we present a pose-invariant shape matching methodology for complete 3D object models. Our approach is based on first describing the objects with shape descriptors and then minimizing the distance between descriptors over an appropriate set of geometric transformations. Our chosen shape description methodology is the density-based framework (DBF), which is experimentally shown to be very effective in 3D object retrieval [1]. In our earlier work, we showed that density-based descriptors exhibit a permutation property that greatly reduces the equivocation of the eigenvalue-based axis labeling and moments-based polarity assignment in a computationally very efficient manner. In the present work, we show that this interesting permutation property is a consequence of the symmetry properties of regular polyhedra. Furthermore, we extend the invariance scheme to arbitrary 3D rotations by a discretization of the infinite space of 3D rotations followed by a nearest neighbor based approximate procedure employed to generate the necessary permutations.