Relational Database Dynamic Load Architecture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale relational database systems face inefficiencies due to the need for terabytes of storage for millions of records, where most queries only access a small fraction, leading to unnecessary storage and retrieval of unused data.
Innovation Solution
A relational database architecture with Dynamic Load capability, where records are stored in external files and loaded into the database only when requested, using a record dispatcher to manage and compress data, and set expiration timestamps for efficient data management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If records are stored in a database for the entire retention period, then query access is always available, but storage requirements increase to terabytes and most records are never accessed
Solution Approach 1:
The system segments records into two locations: external storage for long-term retention and database for active querying. Records are segmented by creating separate data files in external storage and only loading relevant portions into the database when queries occur, resolving the contradiction between maintaining access availability and reducing storage requirements
Solution Approach 2:
The system dynamically loads records into the database based on query requirements rather than maintaining static full database storage. The database schema dynamically adapts by loading only necessary records and columns when queries occur, allowing the system to transition between low-storage external storage and high-access database mode as needed
2Speed
If all records are loaded into the database, then query processing is fast, but retrieval time increases due to transferring large amounts of unused data
Solution Approach 1:
The system applies partial action by loading only the necessary records and columns into the database based on query filters and selection criteria, rather than loading all records. This selective loading approach reduces retrieval time for unused data while maintaining fast query processing for the actual data that needs to be accessed
3Quantity of substance
If records are stored externally and loaded dynamically, then storage requirements are reduced, but system complexity increases with mapping and loading mechanisms
Solution Approach 1:
The system introduces an intermediary layer between external storage and the database that handles the mapping and loading operations. This intermediary component abstracts the complexity of data transfer and mapping, presenting a simplified interface that reduces the apparent system complexity while maintaining the benefits of external storage
Solution Approach 2:
The database schema is designed with multi-functionality to handle both stored records and dynamically loaded records from external storage. The same database structure serves both full database operations and selective loading operations, reducing the need for separate specialized systems and thereby reducing overall complexity
Data Source
AI summary
A relational database architecture with Dynamic Load capability, including a relational database for storing records, a storage unit, a record dispatcher for storing a plurality of objects, each containing at least one record, in the storage unit, and a mapping of the objects to the records which they contain, where the record dispatcher is configured to receive a Dynamic Load Request including a key value of desired records, search the mapping to determine which of the objects in the storage unit contain records that match the key value, and load any of the matching records into the database.