Database Segmentation for Large Data Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face performance issues when managing large data sizes, as they store data as a single block, leading to inefficient searching, retrieval, and display, especially in graphical user interfaces, due to the inability to index and access specific portions of data directly, resulting in high memory usage and slow data loading.
Innovation Solution
The method involves dividing large data into segments smaller than a predetermined threshold, storing the first segment in a main table and subsequent segments in an auxiliary table, with indexes generated and stored in both tables to allow for efficient retrieval and indexing of data, enabling direct access to specific portions without loading the entire dataset.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data of great size are stored in a single field with nvarchar(MAX) data type, then the database can store large amounts of data, but the performance for managing such data (storing, retrieving, searching) is low because they are threaded as a single block of data
Solution Approach 1:
The patent divides large data into multiple segments, each stored in separate records of an auxiliary table. The main table stores only metadata (segment count, total size), while the auxiliary table stores actual data segments with indexes. This segmentation allows the database to manage large data efficiently by operating on smaller individual segments rather than treating the entire dataset as a single block, thus resolving the contradiction between storage capacity and management performance.
2Loss of information
If all data are retrieved from the table of the database first, then the data can be searched, but it is not possible to index directly different portions of the data stored in the table
Solution Approach 1:
The patent creates indexes for each data segment in advance and stores them in the auxiliary table. When a search is needed, the system can directly access the pre-created indexes of specific segments without retrieving the entire dataset. This preliminary indexing action enables direct access to portions of data, eliminating the need to load all data into memory first and significantly reducing search time while maintaining full data accessibility.
3Quantity of substance
If the GUI is programmed to show only a first portion of data at first, then the memory usage is reduced, but all the data stored in the database are retrieved by the application at a time anyway
Solution Approach 1:
The patent extracts the need to retrieve all data by storing only necessary metadata (segment count, total size, current position) in the main table and keeping actual data segments in the auxiliary table. The GUI application only needs to retrieve a small amount of metadata from the main table to determine which segments to display, then selectively retrieves only the required data segments from the auxiliary table using stored indexes. This extraction approach allows the GUI to display portions of data efficiently without loading the entire dataset, reducing memory usage while improving retrieval efficiency.
Data Source
AI summary
A database has a main table and an auxiliary table. Data are stored by: dividing the data into a sequence of segments each having a size smaller than a predetermined threshold; storing a first segment into a first field of a record of the main table; storing an index indexing the data into a second field of the record of the main table; storing the other segments into first fields of corresponding records of the auxiliary table; generating indexes associated to the other segments and storing each index into a second field of the record of the auxiliary table storing the associated segment of the sequence and also in a third field of the record of the auxiliary table storing a preceding segment in the sequence. A third record of the main table stores the index of a first segment of the sequence which is stored in the auxiliary table.