Transparent External Data Access for Memory-Limited Data Servers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data servers with in-memory databases face limitations in data capacity and integration challenges, making it impossible to load large datasets directly into memory, requiring significant computing resources for data access and integration.
Innovation Solution
Implementing transparent access to external database engines via query operations using Structured Query Language (SQL) and Open Database Connectivity (ODBC), allowing data operations to be performed externally, reducing the need to pull entire datasets and enhancing efficiency by filtering and aggregating data on the external database.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is loaded into in-memory database, then data access speed is improved, but memory capacity limitation prevents loading large datasets
Solution Approach 1:
The patent segments data storage between in-memory database (for frequently accessed data) and external data sources (for large-scale storage), allowing the system to handle datasets larger than memory capacity while maintaining fast access speeds for active data
Solution Approach 2:
The patent introduces an intermediary layer that manages data between the in-memory database and external sources, enabling transparent access to external data as if it were in-memory data, thus resolving the capacity-speed tradeoff
2Ease of operation
If entire dataset is pulled for operations, then data manipulation is simplified, but computing resources are significantly consumed
Solution Approach 1:
The patent extracts only the necessary data from external sources through query operations rather than pulling entire datasets, performing filtering, aggregation, and join operations at the source to minimize data transfer and reduce computing resource consumption
Solution Approach 2:
The patent performs data filtering, aggregation, and join operations as preliminary actions at the external data source before data is transferred to the in-memory database, reducing the volume of data that needs to be processed in memory and thereby conserving computing resources
Data Source
AI summary
Disclosed herein are systems and methods that do not require loading data into memory to perform daily activities of an application. The data from an external source can be accessed externally and utilized (for example, in a workbook or via a dashboard, etc.) the same way as in-memory data. This is useful when an external data set is too large to fit in the memory. In this manner, all the data that is needed can be stored in an external table that may be accessed as needed.