In-Memory Data Querying for Low-Latency Field-Level Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face issues of over-fetching and under-fetching due to the fixed structure of REST APIs, leading to inefficiencies in data retrieval and increased latency in real-time applications.
Innovation Solution
Implementing an in-memory database with a custom schema and a data query system, such as GraphQL, to allow for real-time data retrieval and transmission based on user requests, preventing over-fetching and under-fetching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If REST APIs with fixed data structure are used, then data retrieval is simple and consistent, but over-fetching and under-fetching occur leading to data inefficiency
Solution Approach 1:
The patent applies dynamics by transforming the static, fixed data structure of REST APIs into a dynamic, flexible query system. The custom schema and query language allow clients to dynamically specify exactly which data fields they need, adapting the data retrieval structure to match the specific requirements of each request rather than forcing all clients to use the same fixed endpoint structure.
Solution Approach 2:
The patent implements local quality by allowing different parts of the data structure to be selectively accessed based on specific needs. Instead of returning entire data objects or fixed collections, the system enables clients to request only the specific fields they require for their particular use case, making each data retrieval operation tailored to local requirements.
2Reliability
If all user data fields are returned, then complete information is provided, but unnecessary data increases network traffic and processing overhead
Solution Approach 1:
The patent applies the extraction principle by allowing clients to selectively extract only the specific data fields they need from the complete data structure. The custom schema and query system enable precise selection of required fields while leaving unnecessary data behind, rather than transmitting complete data objects that contain both needed and unneeded information.
3Measurement precision
If multiple API requests are made to retrieve related data, then precise data is obtained, but request latency increases
Solution Approach 1:
The patent applies merging by combining multiple data retrieval operations into a single query request. The custom schema allows clients to specify multiple data requirements in one request, and the system retrieves all needed data in a single operation rather than requiring separate requests for each data element, thereby reducing latency while maintaining data accuracy.
4Manufacturing precision
If custom data schemas are implemented, then data retrieval precision is improved, but system complexity increases
Solution Approach 1:
The patent introduces an intermediary layer (the custom schema and query system) that sits between the fixed REST API structure and the client applications. This intermediary translates flexible client data requirements into structured queries that the underlying system can handle, providing precision without requiring fundamental changes to the core system architecture.
Data Source
AI summary
A method for live processing of data is disclosed. The method may include receiving data from a data source. The method may further include storing the data in an in-memory database. The method may further include retrieving the data from the in-memory database according to a custom schema, in response to a request from a user interface; and. The method may further include transmitting, by the computing system, the data to a web client.