Many image segmentation techniques generate sequences of region partitions by tuning parameters that control the level of detail or abstraction, but sequences often violate hierarchical consistency, where each region at one level should be entirely contained within a region at the next. Such inconsistencies undermine the benefits of hierarchical models in multiscale analysis, such as structural coherence and computational efficiency. Despite the importance of hierarchy, existing methods often overlook how to detect and correct violations in segmentation sequences. In this work, we propose a framework for identifying and classifying partitions that most harm the hierarchy. We introduce a recursive measure to quantify the accumulated inconsistency within a sequence and present an optimization strategy that pinpoints and removes the most harmful partition. Experiments on several datasets demonstrate that our approach effectively improves the hierarchiness of the sequence by analyzing and refining the segmentation series.

In the framework of edge-weighted graphs, watersheds have proven to be linked to well-known optimization problems, as Minimum Spanning Tree, which allowed the design of efficient algorithms for computing (hierarchical) watershed segmentations. In the present article, after reviewing the literature related to watershed segmentation, we present a detailed end-to-end pipeline of algorithms to compute (hierarchical) watershed segmentations, starting from the computation of graph-based image representations, up to the computation of connected components of the final (hierarchical) segmentation. We consider the several variations of watersheds, including their supervised and unsupervised versions, and the various ways of computing seeds, to name a few. For the first time, we bring together all these watershed notions and algorithms in a compact and understandable way. We aim at providing a reference for those interested in employing and reimplementing the watershed segmentation framework for their task at hand.

Image segmentation is inherently challenging because it is often difficult to automatically identify the object of interest in an image. To mitigate this, human-provided markers can be incorporated into the segmentation process, greatly improving accuracy. However, human interaction is an expensive resource and methods that reduce effort in interactive segmentation are of great interest. In this work, we introduce a novel marker-based segmentation layer for deep neural networks, enabling end-to-end training of a marker creation network. Our training methodology includes a loss function with two main components: (i) segmentation loss using the new differentiable segmentation layer and (ii) a set of regularization functions that ensure the generated markers have the desired shape properties. We show that by using the proposed method, the network can automatically generate markers that achieve effective segmentation and have desirable shape characteristics. We validate our results in the training dataset and in five unseen datasets.

In this article, we design an incremental method for computing seeded watershed cuts for interactive image segmentation. We propose an algorithm based on the hierarchical image representation called the binary partition tree to compute a seeded watershed cut. Additionally, we leverage properties of minimum spanning forests to introduce a parallel method for labeling a connected component. We show that those algorithms fits perfectly in an interactive segmentation process by handling user interactions, seed addition or removal, in linear time with respect to the number of affected pixels. Run time comparisons with several state-of-the-art interactive and non-interactive watershed methods show that the proposed method can handle user interactions much faster than previous methods with a significant speedup ranging from 10 to 60 on both 2D and 3D images, thus improving the user experience on large images.

Superpixel segmentation partitions multiple objects into disjoint parts so that their delineation can be accurately achieved by their grouping, and it has been used as an intermediary step for solving multiple problems. However, state-of-the-art algorithms face a significant challenge of effective and efficient segmentation irrespective of the problem’s domain (object and background characteristics, and user’s desires). In this work, we address such challenge by proposing several contributions. One of such is a novel superpixel segmentation framework, named Superpixels through Iterative CLEarcutting (SICLE), which generalizes two other contributions of this work. In SICLE, three independent steps are defined: (i) seed oversampling; (ii) superpixel generation using the Image Foresting Transform (IFT) framework; and (iii) seed removal. From (i), where a significantly high amount of seeds is selected, steps (ii) and (iii) are performed for generating superpixels from a refined seed set until achieving the desired number of superpixels. SICLE overcomes domain shifts primarily through steps (ii) and (iii), where the user may provide an objective function for optimization. Experimental results show that SICLE variants surpass several state-of-the-art algorithms concerning speed and accuracy for distinct domains while generating a series of segmentations in a single execution. Still, in SICLE, the contours from a preceding scale might not be present in the subsequent one leading to hierarchical violations. Thus, we studied eight possible cases when analyzing pairwise subsequent segmentations, and we conceived three measures for estimating the hierarchiness of a multiscale segmentation: (i) nestedness; (ii) inflation ratio; and (iii) refinement error. From our results, it is possible to verify if a multiscale is a hierarchy and, when it is not the case, to analyze and state the nature and extent of the hierarchical violations that prevent it from being hierarchical.

This article presents an algorithm to construct the binary partition tree by altitude ordering based on Borvka's minimum spanning tree strategy. The binary partition tree by altitude ordering is a data structure to represent an image in the form of a hierarchy of segmentations that is widely used in mathematical morphology processing. Recently, the authors of PANDORA [21] have proposed a novel parallel algorithm for computing single linkage clustering from the minimum spanning tree of a point cloud on a GPU. In this paper, we show that their method can indeed be nicely cast in the framework of watershed cuts and waterfalls for computing the binary partition tree by altitude ordering. More precisely, the method consists in performing a sequence of watershed cuts-basins contractions, seen as a variant of Borvka's algorithm with a complexity of O(n log(n)), followed by a dedicated post-processing. Furthermore, we show that this algorithm can be extended to process any edge-weighted graphs and not only minimum-spanning trees. These results open a new path towards massively parallel algorithms for hierarchical watershed algorithms.

Numerous segmentation methods are able to produce several partitions of the same image by tuning a scale parameter. In such a series of multilevel segmentations, if every region at a given level is included in a single region of the segmentation at the next level, then the series is called a hierarchy. Hierarchies are often desired for multiscale image representation and analysis due to their mathematical properties, leading to accurate and efficient solutions. Although certain effective strategies may not produce a hierarchy, it is uncertain whether their multiscale output is close to be one. This work explores several cases when analyzing two consecutive segmentations, as full inflation and full merge, for instance. From those, we provide three measures for evaluating the hierarchiness between two subsequent partitions: (i) nestedness; (ii) refinement error; and (iii) inflation ration. Using our proposals in a in-sequence pairwise comparison, as shown by the experimental results, it is possible to verify whether a multiscale segmentation is a hierarchy and, if not, to analyze the nature and extent of the hierarchical errors that prevent it from becoming hierarchical.

Binary Partition Hierarchies (BPHs) and Minimum Spanning Trees are key structures in hierarchical image analysis. However, the explosion in the size of image data poses a new challenge, as the memory available in conventional workstations becomes insufficient to execute classical algorithms. To address this problem, specific algorithms have been proposed for out-of-core computation of BPHs, where a BPH is actually represented by a collection of smaller trees, called a distribution, thus reducing the memory footprint of the algorithms. In this article, we address the problem of designing efficient out-of-core algorithms for computing classical attributes in distributions of BPHs, which is a necessary step towards a complete out-of-core hierarchical analysis workflow that includes tasks such as connected filtering and the generation of other representations such as hierarchical watersheds. The proposed algorithms are based on generic operations designed to propagate information through the distribution of trees, enabling the computation of attributes such as area, volume, height, minima and number of minima.

Hierarchical image segmentation aims to capture the structure of objects of different sizes at different scales and helps to understand the scene. With the success of neural networks for image segmentation and the recent emergence of object and part segmentation datasets, the task of supervised learning of segmentation hierarchies naturally arises. In a previous work, we proposed a differentiable ultrametric layer that transforms any dissimilarity measure into an ultrametric distance equivalent to a hierarchical segmentation. In this paper, we study several loss functions for end-to-end learning of a neural network model predicting hierarchical segmentations. In particular, we propose a generalization of the Rand index for hierarchical segmentation and propose exact and approximate algorithms to compute it. We introduce new metrics to compare hierarchical segmentations, and we demonstrate the suitability of the proposed pipeline with several possible loss function combinations on a simulated hierarchical dataset.

We study the problem of predicting hierarchical image segmentations using supervised deep learning. While deep learning methods are now widely used as contour detectors, the lack of image datasets with hierarchical annotations has prevented researchers from explicitly training models to predict hierarchical contours. Image segmentation has been widely studied, but it is limited by only proposing a segmentation at a single scale. Hierarchical image segmentation solves this problem by proposing segmentation at multiple scales, capturing objects and structures at different levels of detail. However, this area of research appears to be less explored and therefore no hierarchical image segmentation dataset exists. In this paper, we provide a hierarchical adaptation of the Pascal-Part dataset [2], and use it to train a neural network for hierarchical image segmentation prediction. We demonstrate the efficiency of the proposed method through three benchmarks: the precision-recall and F-score benchmarks for boundary location, the level recovery fraction for assessing hierarchy quality, and the false discovery fraction. We show that our method successfully learns hierarchical boundaries in the correct order, and achieves better performance than the state-of-the-art model trained on single-scale segmentations.

Reproducible research in pattern recognition can be viewed from a number of angles, including code execution, platforms that promote reproducibility, code sharing, or the release of libraries providing access to relevant algorithms in the corresponding disciplines. In this work, after recalling the motivation and classic definitions of reproducible research, we propose an updated overview of the main platforms that might be used for reproducible research. We then review the different libraries that are commonly used by the pattern recognition, computer vision, imaging and geometry processing communities, and we share our experience of developing a research library. In the third part, new advanced editorial investments will be presented, such as the IPOL journal or other IPOL-inspired new initiatives like OVD-SaaS.

On the verge of superpixel methods exploiting saliency information, the Superpixels through Iterative CLEarcutting (SICLE) framework has reported fast and accurate superpixel delineation. It is composed of three steps: (i) seed oversampling; (ii) superpixel generation; and (iii) seed removal. It starts from (i) and applies several iterations of (ii) and (iii) until reaching the desired superpixel quantity. In this work, we improve SICLE such that it can now generate compact superpixels with accurate delineation. We exploit differential computation and propose several novel functions for steps (ii) and (iii) for proper saliency incorporation, compact superpixel generation, and improvement in speed and delineation. Results show that, with our proposals, SICLE achieves state-of-the-art performance in delineation and speed whenever saliency is absent with on-par compacity. When an accurate saliency map is provided, its performance improves significantly and requires only two iterations for segmentation.

Most of the time, real-world data sets are composed of classes with the same number of samples. But sometimes, for example in the case of fraud detection or rare disease diagnosis, the resulting dataset is composed of asymmetric classes. These datasets are called imbalanced datasets. Classification problems based on imbalanced datasets lead to errors and high variability. Thus, methods to deal with have been developed. In this paper, we pro-pose a novel oversampling method, called WSSMOTE, based on the water-shed transformation. We demonstrate that WSSMOTE improves prediction scores in some real-world datasets. In addition, our main goal is to improve prediction scores of an imbalanced dataset composed of sickle cell disease (SCD) biomarkers. SCD is a serious-inherited disease, and patients with SCD can be affected by vaso-occlusive crises, which are the main cause of hospitalization. During these hospitalizations, acute chest syndrome (ACS) is the leading cause of death. ACS occurs

In this article, we propose an incremental method for computing seeded watershed cuts for interactive image segmentation. We propose an algorithm based on the hierarchical image representation called the binary partition tree to compute a seeded watershed cut. We show that this algorithm fits perfectly in an interactive segmentation process by handling user interactions, seed addition or removal, in time linear with respect to the number of affected pixels. Run time comparisons with several state-of-the-art interactive and non-interactive watershed methods show that the proposed method can handle user interactions much faster than previous methods achieving significant speedup from 15 to 90, thus improving the user experience on large images.

In this article, we propose a method for designing loss functions based on component trees that can be optimized by gradient descent algorithms and are therefore usable in conjunction with recent machine learning approaches such as neural networks. The nodes of this tree are the connected components of the upper level sets of an image and the leaves represent the regional maxima (or regional minima if the dual tree is considered) of the image, i.e. , connected sets of bright pixels surrounded by darker pixels. The proposed loss function is thus defined at the level of connected components rather than at the level of individual pixels, which allows for the optimization of higher semantic level quantities such as topological features. We show how the altitudes associated with the nodes of such hierarchical image representations can be differentiated with respect to the values of the image pixels. This property is used to design a generic loss function that can select or discard image maxima based on various attributes, such as extinction values based on the contrast or the size of the maxima. The possibilities of the proposed method are demonstrated on simulated and real image filtering.

Binary Partition Hierarchies (BPH) and minimum spanning trees are fundamental data structures involved in hierarchical analysis such as quasi-flat zones or watershed. However, classical BPH construction algorithms require to have the whole data in memory, which prevent the processing of large images that cannot fit entirely in the main memory of the computer. To cope with this problem, an algebraic framework leading to a high level calculus was introduced allowing an out-of-core computation of BPHs. This calculus relies on three operations: select, join, and insert. In this article, we introduce three efficient algorithms to perform these operations providing pseudo-code and complexity analysis.

Superpixels through Iterative CLEarcutting (SICLE) is a recently proposed framework for superpixel segmentation. SICLE consists of three steps: (i) seed oversampling; (ii) superpixel generation; and (iii) seed removal; such that, after step (i), steps (ii) and (iii) are repeated until a desired number of superpixels is obtained. Such pipeline showed effective and efficient multiscale superpixel segmentation. Furthermore, if an object is desired, it is possible to improve delineation by providing its probable location, often called saliency. While classical methods estimate object saliency by contrast-based criteria, recent ones use deep-learning strategies for accurate estimation. SICLE shows robustness for low-quality saliency estimations, but it struggles to effectively take advantage of the high-quality ones. In this work, we propose a generalization of its path-cost function and seed removal criterion (steps (ii) and (iii), respectively), adapting SICLE to a given saliency map. By choice of a binary parameter, SICLE can take advantage of low- and high-quality saliency maps for better segmentation. Results show that, by exploiting the accurate information of the saliency map, our improved SICLE version surpasses state-of-the-art methods in traditional delineation metrics while requiring only two iterations for segmentation, being significantly faster than its predecessor and SLIC.

Image segmentation is an ill-posed problem by definition, as it is not always possible to automatically select which object appearing in an image is the object of interest. To deal with this issue, prior knowledge in the form of human-given markers can be included in the segmentation pipeline. Even though user interaction can drastically improve segmentation results, it is an expensive resource, and finding ways to reduce human effort on an interactive segmentation loop is of great interest. In this work, we propose a new segmentation layer to be used with deep neural networks, which allows us to create and train in an end-to-end fashion a marker creation network. To train the network, we propose a loss function composed of: a segmentation loss using the proposed differentiable segmentation layer; and a set of regularization functions that enforce the desired characteristics on the produced markers. We showed that by using the proposed layer and loss function, we can train the network to automatically generate markers that recover a good segmentation and have desirable shape characteristics. This behavior is observed on the training dataset, as well as on four unseen datasets.

The extension of Mathematical Morphology to colour and multivariate images is challenging due to the need to define a total ordering in the colour space. No one general way of ordering multivariate data exists and, therefore, there is no single, definitive way of performing morphological operations on colour images. In this paper, we propose an extension to mathematical morphology, based on reduced ordering, specifically the morphological Hit-or-Miss Transform which is used for object detection. The reduced ordering employed transforms multivariate observations to scalar comparisons allowing for an order to be derived and for both flat and non-flat structuring elements to be used. We also compare other definitions of the Hit-or-Miss Transform and test alternative colour ordering schemes presented in the literature. Our proposed method is shown to be intuitive and outperforms other approaches to multivariate Hit-or-Miss Transforms. Furthermore, methods of setting the parameters of the proposed Hit-or-Miss Transform are introduced in order to make the transform robust to noise and partial occlusion of objects and, finally, a set of design tools are presented in order to obtain optimal values for setting these parameters accordingly.

This article extends a classical marker-based image segmentation method proposed by Salembier and Garrido in 2000. In the original approach, the segmentation relies on two sets of pixels which play the role of object and background markers. In the proposed extension, the markers are not represented by crisp sets, but by fuzzy ones, i.e., functions of the image domain into the real interval [0, 1] indicating the degree of membership of each pixel to the markers. We show that when the fuzzy markers are indicator functions of crisp sets, the proposed method produces the same result as the original one. We present a linear-time algorithm for computing the result of the proposed method given two fuzzy markers and we establish the correctness of this algorithm. Additionally, we discuss possible applications of the proposed approach, such as adjusting marker strength in interactive image segmentation procedures and optimizing marker locations with gradient descent methods.