Transparent External Data Access via ODBC APIs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data servers with in-memory databases face limitations in data capacity and integration challenges, making it difficult to handle large data sets and requiring significant computing resources for operations like data filtering and aggregation.
Innovation Solution
Implementing transparent access to external databases through APIs such as ODBC, allowing operations to be performed externally, reducing the need to load entire datasets into memory and leveraging external computation power.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is loaded into in-memory database to perform operations, then data processing speed is improved, but memory capacity is exceeded when handling large datasets
Solution Approach 1:
The patent segments data into two parts: data stored in the in-memory database for fast access and data stored in external sources for bulk storage. The system queries external sources only when needed, allowing the in-memory database to remain small and fast while accessing large external datasets on demand.
Solution Approach 2:
The patent introduces an intermediary layer between the in-memory database and external data sources. This intermediary enables the in-memory database to access external data without loading it entirely into memory, resolving the conflict between speed and capacity by allowing selective data access.
2Ease of operation
If entire datasets are loaded into memory for processing, then data availability is improved, but computing resources are overwhelmed
Solution Approach 1:
The patent extracts data processing operations from the in-memory database and relocates them to external sources. Instead of loading entire datasets into memory for processing, the system performs operations directly on external data sources, reducing the computational burden on the in-memory database while maintaining data availability.
Solution Approach 2:
The patent applies partial action by loading only the necessary portions of data into memory rather than entire datasets. The system loads data into the in-memory database only when needed for specific operations, reducing computing resource consumption while maintaining data availability for those operations.
3Productivity
If data filtering and aggregation operations are performed in the in-memory database, then data processing efficiency is improved, but data integration complexity increases
Solution Approach 1:
The patent inverts the traditional approach by performing data filtering and aggregation operations at the external source rather than after loading data into the in-memory database. This reversal of the processing order simplifies data integration by reducing the number of operations needed within the in-memory database while maintaining processing efficiency.
Data Source
AI summary
Disclosed herein are systems and methods that do not require loading data into memory to perform daily activities of an application. The data from an external source can be accessed externally and utilized (for example, in a workbook or via a dashboard, etc.) the same way as in-memory data. This is useful when an external data set is too large to fit in the memory. In this manner, all the data that is needed can be stored in an external table that may be accessed as needed.