External Data Access in Data Servers Without Full Dataset Loading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data servers with in-memory databases face limitations in data capacity and resource-intensive data integration, making it challenging to handle large datasets and perform operations efficiently.
Innovation Solution
Implementing transparent access to external databases via APIs like ODBC, allowing operations such as data filtering, aggregation, and joins to be performed externally, reducing the need to load entire datasets into memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is loaded into in-memory database to perform operations, then operations can be performed efficiently, but memory capacity is exceeded and computing resources are consumed
Solution Approach 1:
The patent segments data storage into two parts: in-memory database for frequently accessed data and external database for large datasets. This allows the system to maintain fast operation efficiency for in-memory data while storing large datasets externally, resolving the contradiction between operation efficiency and data capacity.
Solution Approach 2:
The patent introduces an intermediary layer between the in-memory database and external database. This intermediary enables seamless data access and operations across both storage locations, allowing the system to leverage external storage capacity while maintaining in-memory operation efficiency through controlled data retrieval.
2Speed
If entire datasets are loaded into memory, then data can be accessed quickly, but computing resources related to memory and data integration are excessively consumed
Solution Approach 1:
The patent extracts large datasets from the in-memory database and stores them in external databases. Only necessary data is loaded into memory for operations, significantly reducing memory consumption and computing resources while maintaining fast access speed for the extracted data through efficient querying mechanisms.
Solution Approach 2:
Instead of loading entire datasets into memory, the patent implements partial data loading where only the necessary portions of data are retrieved and processed in memory. This partial action approach reduces computing resource consumption while maintaining adequate data access speed for the actual operations needed.
3Ease of operation
If data operations are performed in the data server, then data can be processed directly, but the data server is constrained by its memory capacity
Solution Approach 1:
The patent extends the data processing capability from a single dimension (in-memory only) to multiple dimensions by incorporating external database access. This dimensional expansion allows the data server to perform operations on data stored externally, effectively increasing the processing volume beyond memory constraints while maintaining ease of operation through unified data access interfaces.
Data Source
AI summary
Disclosed herein are systems and methods that do not require loading data into memory to perform daily activities of an application. The data from an external source can be accessed externally and utilized (for example, in a workbook or via a dashboard, etc.) the same way as in-memory data. This is useful when an external data set is too large to fit in the memory. In this manner, all the data that is needed can be stored in an external table that may be accessed as needed.