In-Memory Schema for Low-Latency Database Query Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database analytic tools are inefficient, costly, and require substantial configuration and training, making it difficult for businesses to access and analyze large volumes of data stored in complex relational database systems.
Innovation Solution
A low-latency database analysis system that uses an in-memory database with a semantic interface to directly execute queries on external data sources, minimizing data movement by generating a schema based on the external database and storing it in the in-memory database, allowing for efficient data analysis without the need to import large volumes of data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional database analytic tools are used to analyze large volumes of data, then data analysis capability is provided, but the tools are inefficient, costly, and require substantial configuration and training
Solution Approach 1:
The system creates a simplified in-memory copy of the external database schema and metadata, allowing complex queries to be executed on this lightweight representation rather than the full external database. This copying approach enables fast analysis without requiring users to directly interact with the complex external database system, thereby improving efficiency while reducing configuration and training needs.
Solution Approach 2:
The system introduces an in-memory database as an intermediary layer between the external database and the user interface. This intermediary handles query processing, data transformation, and result caching, shielding users from the complexity of the external database while providing efficient data access and analysis capabilities.
2Productivity
If large volumes of data are moved from external data sources to the analysis system, then data analysis can be performed, but data movement time and latency increase
Solution Approach 1:
The system extracts only the essential schema, metadata, and query definitions from the external database into the in-memory database, rather than moving large volumes of actual data. This extraction approach allows the system to perform fast in-memory query processing and data manipulation while minimizing data movement time and maintaining low latency.
Solution Approach 2:
The system changes the state of data representation by storing data in optimized in-memory formats with altered parameters (data types, structures, indexing schemes) that are better suited for fast processing. This parameter transformation enables rapid query execution without requiring physical data movement from external sources.
3Quantity of substance
If complex relational database systems are used to store data, then data storage capability is provided, but data accessibility and analysis complexity increase
Solution Approach 1:
The system segments the database functionality into two parts: the external relational database for persistent storage and the in-memory database for rapid access and analysis. This segmentation allows the system to maintain large data volumes in the external database while providing easy accessibility through the simplified in-memory interface, separating storage concerns from access concerns.
Data Source
AI summary
A method and system may be implemented for automatically analyzing data in a database. A method for use in a low-latency database analysis system may include generating a schema. The schema may be based on a portion of an external database. The method may include storing the schema in an in-memory database. The method may include receiving a data-query. The method may include generating a resolved-request. The resolved-request may be based on the data-query and the stored schema. The stored schema may be used for executing the query on the external database. The method may include receiving results data responsive to the data-query from the external database. The method may include outputting the results data for display on a user interface.