Data Search System Using Pre-Computed Views for Low Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional database systems are not scalable to handle large data sets with hundreds of columns and billions of rows, leading to significant latency in query processing, and key-value data stores like Cassandra or HBase cannot process queries with predicates on any data column efficiently.
Innovation Solution
A data search system that utilizes a Kudu data repository, a Parquet data repository, and Spark clustered computing instances managed by a Spark cluster manager, generating pre-computed views in a column-oriented storage format to provide low-latency responses to search queries by storing and retrieving only necessary data columns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional database systems are used to store large data sets with hundreds of columns and billions of rows, then data storage capacity is sufficient, but query processing latency becomes significant and the system is not scalable
Solution Approach 1:
The patent segments the data storage system into multiple specialized components: Kudu tables for real-time data with row and column projections, Parquet tables for batch-optimized storage, and Spark for distributed query processing. Each component handles specific types of data access patterns, allowing the system to scale horizontally while maintaining low query latency through parallel processing across the segmented architecture.
2Loss of time
If key-value data stores like Cassandra or HBase are used, then query processing time improves, but the system cannot process queries with predicates on any data column
Solution Approach 1:
The patent creates a universal data processing system that can handle multiple query types through a unified Spark-based engine. The system generates and stores multiple pre-computed views (row projections, column projections, aggregated views) that can satisfy different query requirements. This multi-functional approach allows the system to process both simple key-value lookups and complex columnar queries with predicates on any column, eliminating the versatility limitation of traditional key-value stores.
3Adaptability or versatility
If traditional database systems process queries on large data sets, then comprehensive query capabilities are maintained, but network traffic and processing overhead increase significantly
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing multiple views of the data in advance. Row projections, column projections, and aggregated views are generated beforehand and stored in optimized formats. When a query arrives, the system can directly retrieve pre-computed results or relevant projections without performing complex computations at query time, significantly reducing network traffic and processing overhead while maintaining comprehensive query capabilities.
Data Source
AI summary
A method for providing low-latency responses to queries to a data search system comprises storing first digital data in a first digital data repository using a first column-oriented storage format; storing second digital data in a second digital data repository using a second column-oriented storage format, wherein the second column-oriented storage format is different from the first column-oriented storage format; generating a plurality of pre-computed views based on both the exposure data and the risk item data; storing the plurality of pre-computed views in the second data repository using the second column-oriented storage format; and in response to receiving a search query, determining a result set that is responsive to the query based on one or more particular pre-computed views of the plurality of pre-computed views and providing the result set to another computer or display device as output.