Hash Pool Family Feature Vector Generation for Data Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data analysis systems face challenges with high computational time due to enormous feature vector dimensions and accuracy deterioration when applying dimension reduction techniques, particularly in handling multiple or real value data items, such as drug efficacy analysis.
Innovation Solution
A data analysis apparatus and method that employs a processor to generate learning feature vectors by using a hash pool family with a-bit and b-bit hash functions, aggregating minimum hash values through odd-filtering or folding calculations, to reduce dimensionality while maintaining analysis accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If hash functions are used to determine minimum hash values for data analysis, then the analysis can be performed, but the number of dimensions of the feature vector becomes enormous (e.g., 429,496,729,600 dimensions with normal parameters), resulting in time-consuming calculation processing
Solution Approach 1:
The patent segments the enormous feature vector into multiple smaller sub-feature vectors, each corresponding to a specific hash function. Instead of processing all 429 billion dimensions at once, the system divides them into manageable chunks that can be processed independently and then aggregated, significantly reducing the time complexity of calculation operations.
Solution Approach 2:
The patent transforms the problem from handling an enormously high-dimensional feature vector directly into a two-stage process: first generating minimum hash values through hash functions, then aggregating these values through a second hash function. This dimensional transformation converts an intractable 429 billion-dimensional problem into a manageable aggregation process with controlled dimensionality.
2Loss of time
If the feature vector is truncated to lower b-bits to reduce the number of dimensions, then calculation processing time is shortened, but the analysis accuracy deteriorates
Solution Approach 1:
The patent performs preliminary action by first computing the minimum hash values through the first hash functions before any truncation or aggregation occurs. This ensures that the essential information is captured in the minimum values before dimensionality reduction, preserving analysis accuracy while enabling subsequent efficient processing through the second hash function aggregation.
Solution Approach 2:
The patent introduces an intermediary aggregation step using a second hash function that combines the minimum hash values from multiple first hash functions. This intermediary process acts as a bridge between the high-dimensional minimum hash values and the final reduced-dimensional feature vector, preserving accuracy information while achieving dimensionality reduction.
3Adaptability or versatility
If traditional hash function methods are used, then binary purchase information can be processed, but the system cannot handle data items with multiple values or real values such as drug prescription amounts
Solution Approach 1:
The patent changes the parameter representation by allowing hash functions to process not only binary values but also multiple discrete values and real-valued parameters. The hash functions are applied to normalized data items regardless of their original type, transforming diverse data types into a unified hash value space that can be aggregated meaningfully.
Solution Approach 2:
The patent creates a universal data analysis framework that handles multiple data types (binary, categorical, real-valued) through the same hash function aggregation process. The system is designed to be multi-functional, accommodating different data item types without requiring separate processing pipelines, thereby increasing versatility while maintaining manageable system complexity through standardized operations.
Data Source
AI summary
A data analysis apparatus executes: acquiring a group of learning input data; setting a plurality of first hash functions; substituting each learning input data in each of the plurality of first hash functions to thereby calculate a plurality of first hash values; selecting, for the group of learning input data and for each of the plurality of first hash functions, a specific first hash value that satisfies a predetermined statistical condition from among the plurality of first hash values; setting a second hash function; substituting, for the group of learning input data, each specific first hash value in the second hash function to thereby calculate a plurality of second hash values; and generating a learning feature vector that indicates features of the group of learning input data by aggregating the plurality of second hash values corresponding to respective specific first hash values and obtained as a result of the calculation.