Bulk Lazy Loading Structured Data Cache
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data loading techniques for structured data objects, such as hierarchical data structures, often result in inefficient database access due to separate retrieval of data by each object, leading to increased database connections and redundant requests, especially when dealing with large datasets or limited database connections.
Innovation Solution
Implementing a method that uses a loaded child record table to identify and cache required data records, allowing for bulk lazy loading by generating a single request for missing records, reducing redundant database queries and improving data access efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If each object separately retrieves data from the database, then data can be loaded on-demand, but the number of database connections and requests increases significantly
Solution Approach 1:
The patent merges multiple separate data retrieval operations into a single bulk retrieval operation. Instead of each object independently querying the database, the system collects all required child records and retrieves them in one consolidated query, reducing the number of database connections from N (where N is the number of objects) to 1.
Solution Approach 2:
The system performs preliminary identification of all required child records before executing the database query. By analyzing the hierarchical data structure in advance and collecting all unique child record identifiers, the system prepares a comprehensive retrieval request that anticipates all data needs, avoiding subsequent separate queries.
2Productivity
If each object loads its required data separately, then data is retrieved when needed, but redundant requests are made for the same data
Solution Approach 1:
The system creates a set of unique child record identifiers by collecting and deduplicating references from multiple parent objects. This copying and consolidation approach identifies that the same child record may be referenced by multiple parents, ensuring it is retrieved only once and then shared across all requiring objects.
Solution Approach 2:
Multiple separate retrieval requests are merged into a single bulk retrieval operation. The system combines all required child record identifiers into one query, eliminating redundant database trips and reducing the total energy consumption associated with repeated data transfer and processing.
3Speed
If all required data is loaded into memory at application start, then data is immediately available, but memory consumption increases for large datasets
Solution Approach 1:
The system performs preliminary bulk retrieval of all required child records in a single operation, preparing the data before it is actually needed by individual objects. This preliminary action loads data efficiently in one batch rather than incrementally, optimizing the balance between availability and resource usage.
Solution Approach 2:
The patent merges the data loading process into a single bulk operation that retrieves all necessary child records simultaneously. This combining approach loads the complete required dataset into memory in one go, ensuring fast access speed while minimizing the total time data resides in partial loading states, thereby optimizing memory utilization patterns.
Data Source
AI summary
Technologies described herein can be used for the bulk lazy loading of structured data from a database. A request can be received to initialize an application data structure (such as a structured data object, a hierarchical data structure, an object graph, etc.). The data structure can be analyzed to identify a plurality of child objects of the data structure. Database records associated with the plurality of child objects can then be identified. A loaded child record table can be inspected to determine which of the identified database records are not stored in a cache. A request can be generated, comprising one or more queries to retrieve the uncached subset of database records from the database. Once the uncached subset of records are received from the database, these records can be used, along with the cached subset of the identified database records, to initialize the plurality of child objects of the application data structure.