Database Slice Pruning via Document Parameter Statistics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database systems face increased load during query processing due to the need to consider large amounts of data, which can be mitigated by reducing the scope of relevant data through pruning, but existing methods are inefficient in identifying and processing only necessary data chunks.
Innovation Solution
A method that analyzes parameters of stored documents across multiple slices in a database, generates statistics on these parameters, and prunes unnecessary slices based on query-specific statistics, allowing only relevant slices to be processed, thereby reducing the load on the system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If more threads are used to process disjunctive pieces of data, then processing time decreases, but system load increases
Solution Approach 1:
The patent extracts and removes irrelevant data slices from the processing scope by applying pruning rules. Statistics about document parameters (min/max values, ranges) are used to identify and exclude slices that cannot contain matching documents, thereby reducing the amount of data that needs to be processed by multiple threads and lowering system load while maintaining processing speed.
Solution Approach 2:
The patent segments the data storage into multiple slices organized in partitions, allowing independent processing of different slice ranges. This segmentation enables selective pruning of entire slices based on statistics, so that only relevant segments are processed by thread pools, reducing both processing time and system load.
2Reliability
If all slices are processed to ensure complete query results, then result accuracy is maintained, but processing efficiency decreases
Solution Approach 1:
The patent performs preliminary analysis of document parameters to generate statistics (min/max values, ranges) for each slice before query processing. These pre-computed statistics are stored and used to quickly determine which slices can be safely pruned, ensuring that no relevant documents are missed while avoiding processing of irrelevant slices, thus maintaining accuracy while improving efficiency.
Solution Approach 2:
The patent uses stored statistics about document parameters as feedback to dynamically determine which slices to process. The pruning rules compare query parameters against the pre-analyzed statistics, providing feedback that guides selective slice processing, ensuring complete results are obtained from relevant slices while skipping irrelevant ones.
3Measurement precision
If statistics are generated and stored for all document parameters, then pruning accuracy improves, but storage requirements increase
Solution Approach 1:
The patent applies local quality by generating and storing statistics selectively for specific document parameters that are relevant to pruning decisions (such as min/max values and ranges of key fields). Rather than storing comprehensive statistics for all possible parameters, the system focuses on essential parameters that enable effective pruning, reducing storage requirements while maintaining pruning accuracy.
Data Source
AI summary
A system for storage, access, and/or pruning of documents of a document store is provided. In some implementations, the system performs operations comprising analyzing a plurality of parameters of a plurality of documents stored among a plurality of slices in a database and generating, based on the analyzing, statistics about the plurality of parameters including a range of document identifiers present in corresponding slices from the plurality of slices. The operations can further comprise pruning, based on the statistics, one or more slices of the plurality of slices, the pruning triggered in response to a query related to at least a portion of the plurality of stored documents. The operations can further comprise responding to the query based on slices that are not pruned. One or more of the operations can be implemented by a database server. Related systems, methods, and articles of manufacture are also described.