Data Processing System Segregating Structured and Binary Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data processing systems face challenges in achieving high-speed processing and permanence of both structured and binary data in IoT applications, particularly with large volumes of sensor data, where index generation becomes a bottleneck and cache capacity limits the effectiveness of data reading and writing.
Innovation Solution
A data processing system comprising first and second data storage devices, along with a controller that determines data type for optimal storage, using a data holding device for high-speed storage and a storage medium for permanence, separating structured and binary data processing to enhance access speed and data permanence.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If indexes are generated for all sensor data to increase reading speed, then data search speed is improved, but the data size of the index exceeds the data size storable in the memory and index generation becomes a bottleneck upon data writing
Solution Approach 1:
The patent segments sensor data into two categories: latest data (frequently accessed) and historical data (less frequently accessed). Indexes are generated only for the latest data stored in memory, while historical data is stored in storage devices without indexes. This segmentation resolves the contradiction by limiting index size to fit memory capacity while maintaining fast access for frequently accessed data.
Solution Approach 2:
The patent applies different data management strategies to different portions of the data based on access frequency. The latest data in memory receives the high-quality treatment of having indexes generated for fast access, while historical data in storage devices uses a simpler storage approach. This local quality differentiation optimizes both reading speed for active data and writing speed for incoming data.
2Speed
If cache is used to hold latest sensor data values to increase reading speed, then data reading speed is improved, but the cache capacity is limited and cannot hold large amounts of sensor data
Solution Approach 1:
The patent segments the data storage hierarchy into memory (for latest data with limited capacity) and storage devices (for historical data with large capacity). This segmentation allows the system to utilize the high-speed access of memory for frequently accessed latest data while offloading less frequently accessed historical data to high-capacity storage devices, effectively resolving the capacity limitation of cache/memory.
Solution Approach 2:
The patent introduces a time-based dimension to data management by distinguishing between latest data and historical data. This temporal differentiation allows the system to optimize memory usage by storing only the most recent data in high-speed memory while maintaining large-scale storage capacity through external storage devices, effectively expanding the usable storage capacity beyond physical memory limits.
3Device complexity
If a single storage system is used to permanently store both structured sensor data and binary image data, then system complexity is reduced, but the file format conversion required for binary data reduces writing and reading speed
Solution Approach 1:
The patent segments the storage system into two independent components: a database for structured sensor data and a file storage system for binary image data. This segmentation eliminates the need for file format conversion by providing native support for each data type in its appropriate storage system, thereby resolving the speed penalty associated with format conversion while maintaining manageable system complexity through clear separation of concerns.
Solution Approach 2:
The patent introduces a data management intermediary that routes different data types to appropriate storage systems. This intermediary layer handles the complexity of managing multiple storage types without requiring the application layer to directly handle format conversions, effectively decoupling the complexity from the data access path and maintaining high writing and reading speeds.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A data processing system according to one embodiment includes first and second data storage devices (62a, 62b) having a storage medium capable of permanently storing data; a data holding device (61) having a storage medium capable of storing the data at a higher speed than the first and second data storage devices (62a, 62b); and a controller (63) that determines the type of data to be processed, and, in the case where the data to be processed is structured data, stores the data to be processed in the data holding device (61), then, collectively, in the first data storage device (62a), and, in the case where the data to be processed is binary data, stores the data to be processed in the data holding device (61) and in the second data storage device (62b).