Database Internal Table Chunked Data Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for transferring large quantities of data from databases, such as using SQL statements or ODBC, are excessively time-consuming, especially when dealing with large data sets.
Innovation Solution
The system generates an internal table and transfers data in chunks, with each chunk containing a portion of a column, optionally compressed using dictionaries and indices, allowing for synchronized or unsynchronized data transfer to ensure efficient data retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional methods (SQL statements or ODBC) are used to transfer data from databases, then data can be transferred using standard protocols, but the transfer process is excessively time-consuming
Solution Approach 1:
The patent divides large data sets into smaller chunks for transfer. Each chunk contains a portion of column data from an internal table, allowing for more efficient processing and transmission. This segmentation enables the system to transfer data in manageable units rather than as a single large block, significantly reducing transfer time.
Solution Approach 2:
The patent performs preliminary actions by generating an internal table containing the data to be transferred before the actual transfer process begins. This internal table is prepared in advance with the specific data subsets that need to be transferred, allowing for optimized chunking and compression operations to occur beforehand, thus speeding up the overall transfer process.
2Quantity of substance
If data is compressed using dictionaries and indices, then data transfer size is reduced, but additional processing overhead is introduced
Solution Approach 1:
The patent changes the state of data by applying compression transformations. It converts raw column data into compressed representations using dictionaries that map original values to compressed forms, and indices that track the mapping relationships. This parameter change reduces the volume of data to be transferred while the system manages the complexity through automated compression algorithms.
Solution Approach 2:
The patent introduces dictionaries and indices as intermediary structures between the source data and the transfer process. These intermediaries enable compression by providing lookup tables that map original data values to compressed representations, reducing transfer volume while maintaining data integrity through the intermediary mapping layer.
3Speed
If data is transferred in chunks with synchronized mode, then complete rows can be formed quickly, but input/output costs increase
Solution Approach 1:
The patent segments data into chunks that are transferred in synchronized mode, where each chunk contains data from multiple columns that can be assembled into complete rows. This segmentation allows the receiving system to reconstruct complete records efficiently by combining corresponding chunks from different columns, achieving fast row formation while managing I/O costs through coordinated transfer.
Data Source
AI summary
A method for transferring data from a database may include receiving, at a database management system, a request to transfer, to a client, at least a portion of data stored at a database coupled with the database management system. The database management system may respond to the request by at least generating an internal table containing at least the portion of the data being transferred to the client. The database management system may further transfer at least the portion of the data to the client by at least sending, to the client, one or more data chunks. Each of the one or more data chunks may include at least a portion of a column of data from the internal table. Related systems and articles of manufacture, including computer program products, are also provided.