Multi-Dimensional Data Storage Optimization via Query Pattern Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage methods are inefficient in handling queries with similar patterns, requiring separate normalization for each query, which consumes time and bandwidth, and are limited to 5 normal forms, failing to adapt dynamically to user demands and access patterns, and do not effectively manage increasing data dimensionality.
Innovation Solution
An optimization system analyzes query patterns to identify data dimensionality, characterizing it into 11 dimensions, and implements strategies such as segmenting data and columns, storing data in remote locations, and fragmenting higher dimensions into smaller ones to optimize storage and access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If separate normalization is carried out for each query, then query-specific data organization is achieved, but time and bandwidth consumption increases
Solution Approach 1:
The system performs preliminary analysis of query patterns and pre-determines normalization strategies before queries are executed. By analyzing query patterns in advance and pre-organizing data structures accordingly, the system avoids the need to perform complete normalization for each query from scratch, thereby reducing execution time while maintaining query-specific organization needs.
Solution Approach 2:
The patent combines multiple query normalizations into a unified approach by identifying common query patterns and applying a single normalization strategy that serves multiple queries simultaneously. This merging of normalization operations reduces redundant processing and decreases overall time and bandwidth consumption while still addressing the specific needs of different query types.
2Device complexity
If existing 5 normal forms are used, then data storage structure is simplified, but adaptability to increasing data dimensionality is limited
Solution Approach 1:
The system dynamically determines the appropriate normalization form and storage structure based on real-time query pattern analysis. Instead of being constrained to a fixed 5 normal form structure, the system adapts its data organization strategy according to the specific requirements of incoming queries, allowing it to handle increasing data dimensionality while maintaining structural simplicity where appropriate.
Solution Approach 2:
The patent introduces a new dimension to data organization by analyzing query patterns and determining optimal storage structures beyond the traditional 5 normal forms. This additional dimension of flexibility allows the system to handle complex multi-dimensional data requirements while maintaining efficiency, effectively extending the normalization paradigm to accommodate modern data storage needs.
3Ease of manufacture
If predefined storage structure is created, then data storage is straightforward, but dynamic adaptation to user demand changes is prevented
Solution Approach 1:
The system transitions from static predefined storage structures to dynamic structures that automatically adapt to changing user demands and query patterns. By continuously analyzing query patterns and adjusting data organization strategies in real-time, the system maintains ease of creation through automated structure determination while achieving dynamic adaptability to evolving data access requirements.
Data Source
AI summary
This technology relates to method and optimization systems for optimizing storage of multi-dimensional data in data storage. The method comprises analyzing a plurality of queries received over period of time from one or more client machines. Then, a query pattern is determined from plurality of queries. Based on query pattern dimensionality of data is identified for determining data storage strategy. The dimensionality is characterized into 11 dimensions comprising 4 standard level dimensions and 7 higher level dimensions. A highest dimension out of 7 higher dimensions is parallel data storage dimension. Based on storage strategy, at least one of data and columns of a table is segmented in data storage. Next, data is stored in remote data storage when data is an element of last higher level dimension. Then, higher level dimensions are fragmented into one or more smaller level dimensions when data is element greater than 11 dimensions.