IO Pipeline Overlay for Parallel Database Segment Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, leading to inefficiencies in data processing and query execution.
Innovation Solution
A database system architecture that employs parallelized data input, storage, retrieval, and query processing systems, utilizing segment-based indexing and optimized query plans to distribute data across multiple computing devices for efficient storage and processing, allowing concurrent query execution and reduced execution times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored and processed using traditional sequential methods in database systems, then hardware resource usage is simplified, but processing speed and execution efficiency are limited
Solution Approach 1:
The patent divides the database system into multiple segments, each handled by separate computing devices. Data is partitioned across these segments, allowing parallel processing of different data portions simultaneously, thereby increasing overall processing speed without requiring a single complex centralized system
Solution Approach 2:
The patent transitions from sequential single-threaded processing to parallel multi-threaded processing across multiple computing devices. This adds a temporal dimension to processing, where multiple operations occur simultaneously rather than one after another, dramatically improving execution speed
2Loss of time
If data is stored in a centralized manner, then system architecture is simpler, but data retrieval and query execution speed are reduced
Solution Approach 1:
The storage system is segmented into multiple distributed storage locations across different computing devices. Each segment stores portions of the database, allowing query results to be retrieved from multiple sources simultaneously rather than sequentially from a single centralized location, reducing overall query execution time
Solution Approach 2:
The patent implements pre-computed indexes and cached query results distributed across the segment architecture. When queries are executed, the system can quickly retrieve pre-prepared data from the appropriate segments rather than processing entire datasets, significantly reducing query execution time
3Productivity
If co-processing options are restricted to maintain system simplicity, then device complexity is reduced, but processing efficiency and throughput are limited
Solution Approach 1:
Each computing device in the segment architecture is designed to perform multiple functions: data storage, query processing, index maintenance, and result caching. This multi-functionality increases processing throughput by utilizing all available resources across all devices rather than dedicating specific devices to single tasks, thereby improving overall productivity without requiring additional specialized hardware
Data Source
AI summary
A database system including a query and response sub-system that includes pluralities of first computing nodes. A set of the first computing nodes implements an input/output (IO) pipeline compiler for a query regarding a dataset, wherein the IO pipeline compiler generates first instructions to create a plurality of IO pipelines for a plurality of segments and second instructions to create a pipeline IO control module. The database system further including a store and computing sub-system that includes a plurality of memory devices that store, in a long-term storage (LTS) format, the plurality of segments and a plurality of processing modules that execute the first instructions to produce the plurality of IO pipelines. The plurality of processing modules further executes the second instructions to produce the pipeline IO control module. The pipeline IO control module controls the plurality of IO pipelines retrieving and converting the plurality of segments from LTS format to a query ready raw data format.