Secondary Data Formats for Multi-Node Query Function Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, leading to inefficiencies in handling large volumes of data.
Innovation Solution
A parallelized database system architecture that includes a parallelized data input, store, retrieve, and process sub-system, along with a query and response system, which optimizes query plans using cost analysis and distributes data storage and processing across multiple computing devices and nodes, allowing for efficient parallel execution of queries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in traditional sequential formats, then storage simplicity is maintained, but processing speed deteriorates due to hardware constraints and inability to parallelize operations
Solution Approach 1:
The patent segments data into multiple formats (primary, secondary, tertiary) with different compression and organization characteristics. Secondary data formats specifically use segmented storage that enables parallel access patterns, dividing large datasets into smaller manageable units that can be processed simultaneously across multiple computing nodes, thereby improving processing speed while managing complexity through structured segmentation
Solution Approach 2:
The patent introduces a new dimension to data storage by organizing data in multi-dimensional structures that enable parallel processing. Secondary data formats utilize columnar organization and hierarchical indexing that add dimensional layers to traditional flat storage, allowing queries to access data from multiple angles and parallelize operations across different data dimensions without proportionally increasing system complexity
2Productivity
If co-processing options are restricted to maintain system simplicity, then device complexity is reduced, but processing capability deteriorates due to limited parallel processing options
Solution Approach 1:
The patent creates a universal co-processing architecture where secondary data formats can be accessed and processed by multiple types of computing nodes (CPU, GPU, FPGA, distributed workers) without requiring format-specific processing logic. This multi-functional design enables diverse processing capabilities to work together on the same data structures, increasing parallel processing capability while reducing the complexity of implementing multiple specialized systems
Solution Approach 2:
The patent introduces secondary data formats as an intermediary layer between raw data storage and processing operations. This intermediary format acts as a universal interface that simplifies co-processing by providing standardized access patterns and metadata structures that different processing units can utilize simultaneously, thereby enhancing parallel processing capability while managing complexity through a unified intermediate representation
3Reliability
If data is processed in real-time to improve response time, then processing speed is improved, but data loss risk increases due to insufficient redundancy and verification time
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing metadata, statistics, and validation information in secondary data formats before actual processing occurs. Checksum fields, data type information, and structural metadata are prepared in advance, allowing real-time processing to proceed quickly while data integrity verification has already been partially completed, thus maintaining both speed and reliability
Solution Approach 2:
The patent implements feedback mechanisms where processing nodes continuously verify data integrity using embedded checksums and validation fields in secondary data formats. Error detection and correction feedback loops operate in parallel with main processing, allowing the system to maintain high processing speeds while ensuring data reliability through continuous verification and automatic correction of detected errors
4Quantity of substance
If massive datasets are processed at scale to improve analytical capability, then data volume handling is improved, but processing time deteriorates due to increased data size
Solution Approach 1:
The patent applies local quality by organizing massive datasets into locally optimized secondary data formats where frequently accessed data regions are stored with lower compression and higher accessibility, while less frequently accessed data uses higher compression. This local optimization allows the system to handle large data volumes by making hot data immediately accessible while maintaining efficient storage for cold data, thereby reducing processing time for active datasets without sacrificing overall data volume capacity
Data Source
AI summary
A method includes ingesting by a set of computing nodes of a parallelized database, a dataset for storage therein. The method further includes formatting, by a lead computing node of the set, the dataset in a primary data format to produce a primary data formatted dataset, and storing the primary data formatted dataset in system state data. The method further includes receiving, by a first computing node of the set, a query request involving the dataset and a first data function, determining that the first data function triggers a first data conversion optimization, accessing the primary data formatted dataset from the system state data, converting the primary data formatted dataset from the primary data format to a secondary data format to produce a secondary data formatted dataset, and processing the secondary data formatted dataset in accordance with the first data function to produce a first function output.