Segmented Storage for High-Dimensionality Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale computing systems face challenges in efficiently storing and retrieving non-sparse high-dimensionality data, such as configuration data for online marketplaces, due to the vast number of potential combinations of values across dimensions, leading to storage inefficiencies.
Innovation Solution
The system partitions the n-dimensional space into non-overlapping segments, with each record representing a shared set of configuration data for contiguous ranges of values, reducing the number of records needed and using ordinal values to organize dimensions based on similarity, allowing for efficient storage and retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If individual records are created for each combination of dimension values in high-dimensionality data, then complete data coverage is achieved, but storage space and record quantity increase significantly
Solution Approach 1:
The patent segments the n-dimensional space into non-overlapping segments (hyperrectangles), where each segment represents a group of contiguous ranges across multiple dimensions. Instead of creating individual records for each dimension combination, a single record stores configuration data applicable to all points within its segment, dramatically reducing the total number of records while maintaining complete data coverage.
Solution Approach 2:
The patent transforms the problem from storing data in a high-dimensional sparse space to organizing it in a segmented hierarchical structure. By introducing the concept of segments that span multiple dimensions simultaneously, the system effectively adds a new organizational dimension, allowing efficient representation of n-dimensional data without exponential growth in record quantity.
2Ease of operation
If individual records are created for each combination of dimension values, then data retrieval is straightforward, but query efficiency deteriorates due to large data volume
Solution Approach 1:
By organizing data into non-overlapping segments that represent contiguous ranges across dimensions, the patent enables efficient range queries. When a query requests data for specific dimension ranges, the system can identify and retrieve only the relevant segments rather than scanning through millions of individual records, significantly improving query efficiency while maintaining ease of retrieval through the structured segment organization.
3Reliability
If configuration data is stored for all potential dimension combinations, then complete coverage is achieved, but storage space is wasted due to redundancy
Solution Approach 1:
The patent merges configuration data that applies to multiple dimension combinations into single segment records. By identifying contiguous ranges across dimensions where the same configuration data is valid, the system combines what would otherwise be multiple separate records into one, eliminating redundancy while maintaining complete coverage. Each segment record represents a consolidation of configuration data applicable to all points within its multi-dimensional bounds.
Data Source
AI summary
Methods, systems, and computer-readable media for compact storage of non-sparse high-dimensionality data are disclosed. A data store comprises a plurality of records, and the plurality of records represent a space having n dimensions of ordered input values. The space is partitioned into a plurality of non-overlapping segments having the n dimensions. Individual ones of the records correspond to individual ones of the segments. A query is received that comprises one or more input values for at least some of the n dimensions of the space. At least one record is retrieved from the data store based at least in part on the query. The record corresponds to one of the segments in the space, and the record comprises content responsive to the query.