Sparse Data Storage in Search Index via Integer Arrays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems for storing sparse data in search indices create a high number of columns, which negatively impact performance and capacity, as only a small percentage of entities have corresponding data points, leading to inefficient use of resources.
Innovation Solution
Implementing an integer array within a single column in the search index to store sparse data points, including an identifier and a value associated with the data point, and transforming queries based on the identifier to provide user-compatible results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If two columns are created in the search index for every entity to accommodate match count and match confidence value, then the data points can be stored and queried, but the search index includes a high percentage of columns for a very small amount of data, negatively impacting performance and capacity
Solution Approach 1:
The patent combines multiple entity-specific columns into a single consolidated column that stores sparse data points. Instead of creating separate columns for each entity's match count and confidence value, the invention merges these into one column that can accommodate data points from multiple entities, thereby reducing the total number of columns while maintaining query capability.
Solution Approach 2:
The patent transitions from a wide-schema approach (many columns) to a long-schema approach (fewer columns with more rows). By storing sparse data in a vertical format within a single column rather than distributing data across multiple horizontal columns, the invention changes the dimensional organization of data, reducing column count while preserving data accessibility.
2Loss of information
If two columns are created for every entity, then match count and confidence values can be stored, but only about 1-2% of entities have corresponding data points, leading to inefficient use of resources
Solution Approach 1:
The patent extracts only the necessary data points from the sparse dataset and stores them in a dedicated column, rather than allocating space for all possible entities. By taking out and storing only the actual data points that exist (1-2% of potential entities), the invention eliminates wasted memory space while ensuring complete storage of relevant information.
Solution Approach 2:
The patent changes the storage parameter from a fixed wide schema with predetermined columns for each entity to a dynamic structure where only actual data points are stored. This parameter change allows the system to adapt to the sparsity of the data, storing information efficiently without pre-allocating memory for non-existent data points.
3Adaptability or versatility
If many columns are created in the search index to accommodate all entities, then all potential data points can be queried, but each additional column negatively impacts system performance and capacity
Solution Approach 1:
The patent creates a single column that serves multiple functions by storing data points from multiple different entities. This universal column can accommodate match count and confidence values for any entity that has sparse data, providing the same query coverage as multiple entity-specific columns would have offered, while improving system performance through reduced column count.
Data Source
AI summary
Sparse data may be stored in a search index such that one or more data points related to a number of entities may be queriable. Upon detection of a data point related to an entity, an integer array may be created for the data point to store within a single column of the search index, the integer array including an identifier of the entity and a value associated with the data point. In response to receiving a request from a user to perform a query on the search index that includes the data point, the query may be transformed to a corresponding range based on the identifier of the entity. Upon execution of the query, the identifier may be removed and query results may be provided to the user as another array with a user-compatible identifier for the entity such that the results may be understood by the user.