Dynamic Storage Format Selection for Sensor Data Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face challenges in efficiently storing and querying large volumes of data from continuous sensor feeds, as current solutions like network package capture tools result in datasets that are difficult to filter and process.
Innovation Solution
A method and system for dynamically determining a storage format from an input data stream by analyzing characteristics and applying predetermined rules to create data packages with metadata, allowing for flexible and efficient storage and retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If network package capture tool is used to store packages, then data storage is achieved, but the dataset cannot be easily queried or filtered
Solution Approach 1:
The patent applies preliminary action by extracting and storing metadata characteristics from raw data subsets before the actual data storage. This metadata includes characteristics that enable easy querying and filtering, so when queries are executed later, the system can quickly locate and filter data packages without having to process the entire dataset, thus resolving the contradiction between storage capability and query ease
Solution Approach 2:
The patent introduces metadata as an intermediary between the raw data packages and the query operations. This metadata layer acts as a mediator that enables efficient querying and filtering without requiring direct access to the raw data, thus maintaining storage reliability while dramatically improving querying and filtering ease
2Reliability
If raw data stream is stored without processing, then data integrity is maintained, but computational overhead increases
Solution Approach 1:
The patent applies the extraction principle by separating metadata characteristics from the raw data stream. Only the essential characteristics are extracted and stored, while the bulk raw data is stored minimally. This reduces computational overhead for storage and retrieval operations while maintaining data integrity, as the full raw data can still be recovered when needed
3Ease of manufacture
If fixed storage format is used, then storage simplicity is maintained, but adaptability to changing data streams is reduced
Solution Approach 1:
The patent applies dynamics by making the storage format adaptive rather than fixed. The system dynamically determines storage formats based on the characteristics of incoming data subsets. This allows the storage system to automatically adapt to different data types, formats, and characteristics while maintaining a consistent overall storage structure, thus resolving the contradiction between storage simplicity and format adaptability
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
A computer-implemented method of dynamically determining data packages of an input data stream, the method comprising: receiving an input data stream comprising a raw data subset; determining characteristics of the raw data subset; based on the characteristics and a predetermined set of rules, determining a storage format for one or more data packages obtainable from the raw data subset; and storing the one or more data packages according to the storage format.