Dynamic Block Size Estimation for Statistical Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The speed of statistical processing is influenced by block size and cannot be effectively increased when an inappropriate block size is selected, leading to inefficiencies in data processing.
Innovation Solution
An information processing device and method that estimates an optimal block size based on query history to minimize the average cost of deriving answers, dividing the data set into blocks and calculating statistical values for each block to enhance processing speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a fixed block size is used for dividing the data set, then the statistical processing can be performed using pre-calculated statistical values, but the processing speed cannot be effectively increased when the block size is inappropriate
Solution Approach 1:
The patent applies dynamics by making the block size adjustable rather than fixed. The system dynamically determines the optimal block size based on query characteristics and historical data, allowing the block configuration to adapt to different processing scenarios and maximize processing speed for various query types
Solution Approach 2:
The patent changes the parameter of block size based on different conditions. By analyzing query patterns and data characteristics, the system modifies the block size parameter to optimize statistical processing performance, transforming a static parameter into a condition-dependent variable
2Loss of time
If the block size is increased to reduce the number of blocks, then the calculation overhead per block increases, but the number of I/O operations decreases
Solution Approach 1:
The patent applies partial action by calculating statistical values for only the necessary portions of data based on query requirements. Instead of processing entire large blocks uniformly, the system selectively processes relevant data portions, reducing unnecessary calculation energy while maintaining efficient I/O operations
3Measurement precision
If the block size is decreased to increase the number of blocks, then the granularity of statistical values increases, but the number of I/O operations increases
Solution Approach 1:
The patent applies local quality by optimizing block size for specific query types and data regions. Different blocks can have different sizes tailored to their characteristics and query requirements, allowing high precision where needed while minimizing I/O operations through larger blocks in other regions
Data Source
AI summary
An information processing device includes a statistical value holding unit configured to hold a statistical value of data included in each of two or more blocks into which a data set is divided; a query history holding unit configured to hold information about a past query on the data set as a query history; an estimation unit configured to, based on the query history, estimate a block size that minimizes an average cost of deriving an answer to a query by using the statistical value of at least one of the blocks; and a block creation unit configured to, based on the block size estimated by the estimation unit, create two or more blocks by dividing the data set, calculate the statistical value for each of the created blocks and cause the statistical value holding unit to hold the calculated statistical values.