Clustering Module Matrix Block Caching for Memory Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional clustering algorithms face inefficiencies due to the need for converting relational tables into plain file or table format, which is time-consuming and wasteful of resources, especially when dealing with large datasets or limited memory systems, and storage requirements are inefficient due to zero-value dimensions.
Innovation Solution
The implementation of a database system with a clustering module that uses matrix manipulation techniques, allowing for efficient processing and reduced memory usage by caching selected blocks of matrices from persistent storage, eliminating the need for full matrix conversion and optimizing storage and processing resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional clustering algorithms use plain file or table format, then data can be processed by clustering algorithms, but conversion from relational tables is time-consuming and wasteful of system resources
Solution Approach 1:
The patent extracts and eliminates the unnecessary conversion step from the data processing pipeline. By allowing clustering algorithms to operate directly on relational table data without converting to plain file format, it removes the time-consuming conversion process while maintaining clustering functionality.
Solution Approach 2:
The patent makes the clustering algorithm more universal by enabling it to work with multiple data formats simultaneously - both traditional plain file formats and relational table formats. This multi-functionality allows the algorithm to choose the most efficient input format based on the data source.
2Reliability
If zero-value dimensions are stored in plain file or table format, then complete data structure is maintained, but storage space and processing resources are wasted
Solution Approach 1:
The patent extracts and removes zero-value dimensions from the stored data structure. By not storing dimensions with zero values, it reduces storage space requirements and processing overhead while maintaining the integrity of meaningful data through appropriate data structures that can represent sparse information efficiently.
Solution Approach 2:
The patent applies different storage strategies to different parts of the data structure. Instead of uniformly storing all dimensions, it selectively stores only non-zero dimensions, applying local optimization to reduce overall storage requirements while maintaining data completeness where needed.
3Reliability
If large datasets with many dimensions are stored completely, then all data is available for processing, but memory may not have sufficient space to store clustering results and intermediate data structures
Solution Approach 1:
The patent segments the large dataset into manageable blocks or partitions that can be processed separately. By dividing the data into smaller chunks, it enables processing of large datasets with limited memory, allowing clustering operations to be performed on subsets of data while maintaining overall data availability through systematic data management.
Solution Approach 2:
The patent processes data in partial iterations rather than requiring complete data loading. By performing clustering operations on subsets of data multiple times or using sampling techniques, it achieves meaningful clustering results without requiring all data to be simultaneously present in memory.
Data Source
AI summary
Matrices involved in a data analysis are stored in predetermined blocks, where blocks for a first matrix contain respective rows of the first matrix, and blocks for a second matrix contain respective columns of the second matrix. Results for the data analysis are computed using the blocks of the first and second matrices.