Formatted Record Processing with Compressed Variable-Length Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, leading to inefficiencies in data handling and query execution.
Innovation Solution
A parallelized database system architecture that includes a parallelized data input sub-system, data store and process sub-system, query and response sub-system, and administrative and configuration sub-systems, utilizing a segmenting and restructuring approach to optimize data storage and query processing across multiple computing devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a traditional database system processes data sequentially with single-threaded operations, then the system structure remains simple, but the data processing speed is limited by hardware constraints and execution time increases
Solution Approach 1:
The patent divides the database system into multiple independent segments that can process data concurrently. Each segment handles specific portions of data or queries independently, enabling parallel processing that increases data processing speed while maintaining manageable complexity through modular architecture
Solution Approach 2:
The patent transitions from single-threaded sequential processing to multi-threaded parallel processing by adding a temporal and structural dimension to data handling. Multiple threads execute queries and operations simultaneously across different data segments, effectively increasing processing speed without proportionally increasing perceived system complexity
2Productivity
If data is stored in traditional fixed-format records, then storage structure is simple, but processing efficiency is limited by rigid schema constraints and inability to handle variable-length fields efficiently
Solution Approach 1:
The patent introduces dynamic record formats that can adapt to variable-length fields and different data types during processing. Instead of rigid fixed-format records, the system dynamically adjusts record structures based on the data being processed, enabling efficient handling of diverse data while maintaining processing speed through optimized variable-length field management
Solution Approach 2:
The patent changes the storage parameters from fixed-length records to variable-length records with flexible formatting. This allows the system to optimize storage and processing by adjusting record parameters dynamically, improving productivity for diverse data types while managing structure complexity through standardized variable-length protocols
3Loss of time
If the database system uses traditional single-threaded query execution, then the implementation is straightforward, but execution time increases and concurrency is restricted
Solution Approach 1:
The patent segments query execution into multiple threads that can run concurrently. Each thread handles specific query operations or data access patterns independently, reducing overall execution time by eliminating sequential bottlenecks while managing complexity through thread-safe design and coordinated access protocols
Solution Approach 2:
The patent ensures continuous productive operation by implementing overlapping query executions where multiple threads work on different data segments simultaneously. This eliminates idle waiting time and maintains continuous useful action across the system, reducing execution time while managing concurrency complexity through efficient thread coordination
Data Source
AI summary
A record processing and storage system operates by: receiving formatted row data that includes a fixed-length batch section and a temporary variable-length batch section; storing, in a compressed data format, at least one variable-length data value of each of a plurality of temporary variable-length row sections of each of the plurality of formatted row data; generating a plurality of modified formatted row data from the plurality of formatted row data by, for each formatted row data, replacing a fixed-length offset value for each variable-length field stored in the compressed data format with a key value utilized to access the corresponding variable-length data value in the compressed data format; and truncating the each formatted row data to remove inclusion of the temporary variable-length batch section; and storing the plurality of modified formatted row data as a plurality of pages in a page storage system.