Lightweight Indexing for Append-Only IoT Data Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge lies in efficiently storing and searching large amounts of data generated by IoT devices, particularly 'cold data' that loses relevance over time, requiring space-efficient and searchable storage solutions as IoT adoption increases.
Innovation Solution
A method involving storing data records as segments with descriptors, employing user-defined functions to resolve minimum and maximum descriptors, and composing a lightweight index for each segment, which is appended to the storage element, enabling efficient querying and retrieval of relevant data segments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in append-only format for space efficiency, then storage density is improved, but searchability and query performance deteriorate
Solution Approach 1:
The patent segments data into fixed-size data blocks with embedded lightweight indices. Each data block contains compressed data segments organized in append-only format along with a lightweight index structure that enables efficient querying without requiring full data block decompression. This segmentation allows the system to maintain high storage density while providing fast search capabilities through the embedded indices.
Solution Approach 2:
The patent introduces lightweight indices as an intermediary layer between the append-only data storage and query operations. These indices contain minimal metadata (such as minimum and maximum values for queryable fields) that mediates between the compact stored data and search requests, enabling fast query filtering without accessing the full data blocks and thus maintaining both storage efficiency and query performance.
2Productivity
If all data is kept in searchable format, then query performance is improved, but storage space consumption increases
Solution Approach 1:
The patent applies local quality by storing full indexing information only for recently accessed or relevant data blocks, while older or less relevant blocks use compressed append-only storage with minimal metadata. The system dynamically adjusts the indexing level based on data age, access patterns, and relevance, allowing fast query performance for hot data while maintaining space efficiency for cold data archives.
Solution Approach 2:
The patent changes the indexing parameter density based on data characteristics and query patterns. For data blocks requiring frequent queries, full indices are maintained; for archival data, minimal metadata is stored. The system adjusts compression levels, index granularity, and metadata retention parameters dynamically to optimize the balance between storage space and query performance for different data subsets.
Data Source
AI summary
A method comprising receiving a plurality of data records, storing the data records as data segments in a storage element, receiving a plurality of descriptors for each data segment, wherein each descriptor describes an aspect of data contained in the data segments, employing a first user-defined function to resolve a first minimum descriptor for each data segment and a first maximum descriptor for each data segment, composing a lightweight index for the data segments, wherein the lightweight index comprises the first minimum descriptor for each data segment and the first maximum descriptor for each data segment, and appending the lightweight index to the data segments in the storage element.