Analytical Storage Filtering With Bitmap-Based Data Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Integrating analytical data processing capabilities into storage systems increases technological complexity, hardware design, and software development costs, and can lead to layer violations and compatibility issues, limiting widespread adoption.
Innovation Solution
The techniques involve receiving commands defining filter criteria and address ranges, identifying data elements that match these criteria, and providing a count or bitmap of these elements, reducing the amount of data transferred to the host computer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If analytical data processing capabilities are integrated into storage systems, then data processing efficiency is improved, but device complexity increases
Solution Approach 1:
The patent segments the data processing function into two parts: the storage device performs initial filtering and bitmap generation, while the host computer performs final data retrieval and processing. This segmentation allows analytical processing capabilities to be added to storage systems without requiring full integration of complex processing units, thereby improving data processing efficiency while controlling device complexity.
Solution Approach 2:
The patent introduces a bitmap as an intermediary data structure that bridges the storage device and host computer. The bitmap efficiently represents filtered data locations without transferring actual data values, enabling analytical processing at the storage layer while maintaining simple storage device architecture. This intermediary approach resolves the contradiction by enabling sophisticated processing through a lightweight mediation mechanism rather than complex integrated processing units.
2Productivity
If analytical data processing capabilities are integrated into storage systems, then data processing speed is improved, but software development costs increase
Solution Approach 1:
The patent uses a bitmap as a simplified copy or representation of the actual data structure. Instead of developing complex processing software that manipulates full data structures, the system uses bitmaps that require minimal software logic to generate and interpret. This copying approach accelerates data analysis by enabling efficient filtering operations while significantly reducing software development complexity and costs.
3Ease of operation
If data is transferred from storage device to host computer, then data accessibility is improved, but data transfer volume increases
Solution Approach 1:
The patent extracts only the essential information needed for data accessibility - specifically, the bitmap that indicates which data elements match filter criteria. Instead of transferring all stored data or even all filtered data to the host computer, the system extracts and transfers only the compact bitmap representation, which uses minimal space while maintaining full accessibility to the relevant data elements through subsequent host-side operations.
Data Source
AI summary
Techniques for providing analytical storage systems and devices capable of executing analytical data operations “near-storage.” The techniques include, upon receipt of a first command from a host computer, identifying which data elements within specified address ranges of selected columns of a data array match defined filter criteria. The techniques include providing, to the host computer, a count of the identified data elements, and/or a bitmap containing binary values indicating locations of the identified data elements in the selected columns. The techniques include, upon receipt of a second command from the host computer, providing, to the host computer, the identified data elements from the column locations indicated by the bitmap. The techniques can reduce, in a single pass, amounts of data to be provided to a host computer during execution of analytical data operations “near-storage.” The techniques can be implemented in systems and devices with reduced technological complexity and simplified programmability.