Storage Array Indexer for Accelerated Analytic Query Execution

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional data storage systems face inefficiencies in processing analytic queries due to the need for multiple round trips between external query processing engines and storage arrays, leading to slow execution and high IO capacity usage, which precludes their use in production OLTP systems and introduces latency, consistency issues, and increased costs.

Innovation Solution

Implementing an indexer within the storage array to generate indexes over stored data, allowing query processing to occur within the storage array itself, utilizing high-speed memory and storage tiering techniques to accelerate query execution and reduce latency.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If query processing is performed externally using a separate database management system, then query processing capability is provided, but query execution speed is slow and IO capacity is excessively consumed

Engineering Contradiction:

Improvequery execution speedVSAvoidIO capacity consumption

Core Design Contradiction:

ProductivityVSLoss of energy

Solution Approach 1:

The patent merges the query processing functionality with the storage array by integrating an indexer and query processing engine directly into the storage system. This allows queries to be executed using indexes stored in the same array, eliminating the need for external data movement and reducing IO capacity consumption while improving query execution speed.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The patent segments the storage array into data storage regions and index storage regions, allowing simultaneous optimization of both data retention and query processing. The index structures are maintained separately from the primary data, enabling efficient query execution without impacting the main data storage operations.

Inventive Principle:

Principle #1Segmentation

2Reliability

If data is extracted to external warehouses for analysis, then analytic queries can be executed, but latency is introduced and data consistency issues arise

Engineering Contradiction:

Improvedata consistencyVSAvoidquery latency

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

By combining the analytical query processing capabilities directly within the storage array, the system eliminates the data extraction and ETL processes that cause latency and consistency issues. Queries access the same data source that maintains the authoritative copy, ensuring consistency while reducing the time required to retrieve and analyze data.

Inventive Principle:

Principle #5Merging (Combining)

3Productivity

If multiple round trips are made between external query engine and storage array, then query processing is completed, but execution time increases significantly

Engineering Contradiction:

Improvequery processing throughputVSAvoidquery execution time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by pre-building and maintaining index structures within the storage array before queries are executed. These indexes are continuously updated as data changes, so when queries arrive, they can be processed immediately using the pre-computed indexes rather than requiring multiple round trips to retrieve and process raw data.

Inventive Principle:

Principle #10Preliminary action

4Speed

If indexes are stored in high-speed memory within the storage array, then query execution is accelerated, but memory capacity requirements increase

Engineering Contradiction:

Improvequery execution speedVSAvoidmemory capacity

Core Design Contradiction:

SpeedVSQuantity of substance

Solution Approach 1:

The patent applies local quality by storing indexes in high-speed memory specifically within the storage array where they are needed for query processing, rather than using uniform storage across the entire system. This localized approach to high-speed storage provides query acceleration exactly where it is most beneficial, while avoiding the need to upgrade the memory capacity of the entire system.

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS9665621B1Accelerated query execution within a storage array

Publication Date: 2017.05.30 EMC IP HLDG CO LLC

AI summary

A data storage system comprises a storage array and an associated indexer that may be implemented in whole or in part within the storage array. The storage array comprises storage devices and a controller. The indexer associated with the storage array is configured to generate indexes over data stored in the storage array. The controller comprises a query processing control point configured to utilize the indexes generated by the indexer to accelerate execution of queries directed to the data stored in the storage array. The indexes may be stored in a high-speed memory of the storage array, such as flash memory. The queries executed in the storage array may comprise analytic queries that would otherwise normally be executed in an analytic data store external to the storage array. By providing index-based processing of the queries within the storage array, query processing throughput of the data storage system is substantially improved.