Non-Relational Database Sort Key Timestamp Querying
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current transaction processing systems face inefficiencies in retrieving and managing large volumes of transaction data, leading to resource-intensive operations and poor user experiences due to slow data retrieval, especially when generating reports or analyzing data for suspicious behavior.
Innovation Solution
Implementing a database management system that uses sort keys, such as timestamps, to store and retrieve transaction data in a non-relational database like Apache Hbase, allowing for efficient horizontal partitioning and quick data retrieval, enabling continuous updates without performance degradation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If transaction data is stored in a traditional relational database, then data integrity and relationships are maintained, but data retrieval becomes slow and resource-intensive when dealing with large volumes of transaction data
Solution Approach 1:
The patent segments the database into multiple tables (transaction table, account table, linking table) with specific schemas optimized for different query patterns. Each table is designed with particular columns and data types to facilitate efficient retrieval for specific analytical purposes, resolving the contradiction by organizing data in a segmented manner that balances retrieval speed with manageable complexity.
Solution Approach 2:
The patent introduces a new dimensional organization by creating specialized views and tables that aggregate transaction data from multiple perspectives (by account, by time period, by transaction type). This dimensional change allows rapid retrieval along different query axes without requiring complex joins across the entire transaction history, thus improving speed while maintaining manageable database complexity.
2Productivity
If all transaction data is loaded into memory for analysis, then quick access and manipulation is achieved, but system memory resources are exhausted and performance degrades
Solution Approach 1:
The patent performs preliminary aggregation and pre-computation of transaction statistics during data ingestion and storage phases. Summary tables and cached aggregates are prepared in advance, allowing subsequent analytical queries to operate on pre-processed data rather than raw transaction records. This preliminary action enables efficient data processing while consuming minimal memory resources during query execution.
Solution Approach 2:
The patent creates simplified copies and projections of transaction data in optimized formats for analytical purposes. Rather than loading complete transaction records into memory, the system generates condensed representations containing only the necessary fields for specific analyses, thereby maintaining high productivity while dramatically reducing memory resource consumption.
3Reliability
If frequent updates are made to transaction data, then data currency and accuracy are maintained, but database performance degrades and retrieval speed decreases
Solution Approach 1:
The patent implements periodic batch processing for data updates and aggregations rather than continuous real-time processing. Transaction data is updated and aggregated at scheduled intervals, allowing the system to maintain data accuracy and currency while avoiding the performance degradation associated with frequent incremental updates. This periodic action preserves retrieval speed by minimizing the frequency of disruptive update operations.
Data Source
AI summary
Techniques are disclosed for storing and retrieving large amounts of data in a non-relational database using sort keys. A server computer system may receive a request for raw data specifying a start timestamp and an end timestamp. The server determines a start key and an end key for performing a query on a distributed non-relational database storing key-value pairs, where the determining is based on the start timestamp and the end timestamp. The server may compare the start key and the end key to a sort key included in row keys of key-value pairs stored in the non-relational database. Based on the comparing, the server retrieves one or more rows of raw data from the non-relational database. The server generates a graphical representation of the one or more rows of raw data retrieved from the non-relational database. The disclosed techniques may advantageously improve the efficiency of a database management system.