Index File Segmentation for IoT Data Write Speed
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional databases face challenges in efficiently writing and reading large amounts of IoT data, particularly in systems where data is generated continuously, as they require extensive indexing, leading to increased processing overhead and difficulty in managing varying data sizes and characteristics, such as images and videos.
Innovation Solution
An information accumulation apparatus that records index information including time instants of data generation and position information in an index file, allowing for efficient data sequencing and correlation with actual data files, minimizing write processing and enabling high-speed read/write operations by using a common database.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If conventional databases use extensive indexing to improve read speed, then data search and retrieval performance is improved, but processing overhead during data write operations increases significantly
Solution Approach 1:
The patent segments the database into two distinct components: a time-series database for efficient sequential write operations and a relational database for complex queries. This segmentation allows each database type to optimize for its specific use case, with the time-series database handling high-volume IoT data ingestion without the overhead of complex indexing structures
Solution Approach 2:
The patent introduces a data management system that acts as an intermediary layer between the time-series database and the relational database. This intermediary manages data distribution, indexing strategies, and query routing, allowing the system to maintain high write speeds while providing efficient read access through intelligent data placement and indexing only where necessary
2Measurement precision
If the number of indices generated for data increases to improve search performance, then data extraction speed is improved, but the processing load and memory usage during write operations increases
Solution Approach 1:
The patent applies local quality by implementing selective indexing strategies where only specific high-value columns and fields are indexed in the relational database portion, while the time-series database maintains data in an optimized sequential format. This localized indexing approach provides sufficient search accuracy for common queries while minimizing the memory overhead and processing load associated with maintaining extensive index structures
3Adaptability or versatility
If conventional databases are used to manage varying data sizes including images and videos, then data storage capability is improved, but device complexity and processing overhead increases
Solution Approach 1:
The patent segments the storage system into a time-series database for structured IoT sensor data and a relational database for unstructured media files and metadata. This segmentation allows each storage component to be optimized for its specific data type, reducing the complexity that would arise from attempting to manage all data types within a single database system
Solution Approach 2:
The patent creates a universal data management architecture that can handle both time-series sensor data and unstructured media files through a unified interface and common storage infrastructure. The system provides multi-functional capabilities including data ingestion, storage, querying, and retrieval across different data types without requiring separate specialized systems for each data category
Data Source
AI summary
When data generated by devices (30) arrive in an order of time instants of generation of the data, the data are additionally described (written) to an actual data recording unit (12b) in an order of arrival of the data, and, meanwhile, a time instant, at which the data additionally written to the actual data recording unit (12b) is generated, a data size, and a position of postscript in an actual data file are additionally written, as index information, to an index file (12a) having a file name corresponding to the actual data file in which the data is recorded.