Efficiently indexing with offset bitmaps for huge sets of slightly disordered sensor data

With advances in sensor devices and networking technologies, it is expected that future networks will contain immense numbers of sensors that are capturing time-varying data. It is necessary to process queries over the data for analysis and to store the data for later use. For querying the data, data disorder is a common problem. Existing approaches use buffers to recover the order but there are problems in deciding buffer sizes and buffer residence times. To avoid these problems and to satisfy the need to store the data for later use, our approach writes indexed data rapidly to a stable disk. This approach, however, has a problem in that existing indexing methods are not up to the task; “sparse indexing” offers fast write-speeds but cannot handle disordered data, “dense indexing” can handle disordered data but its write-speed is inadequate. Our solution is a novel indexing method that can rapidly store slightly disordered time-series sensor data. Our proposal succeeds because it uses a novel data structure, “offset bitmaps”, to extend the sparse indexing method. The offset bitmaps provide pointers that can handle delayed data and so allow them to be managed efficiently. Experiments show that our method has several advantages over conventional alternatives. We verify that the proposed method has 10 times the write speed of the existing dense indexing method.