Shared Memory Parallel Data Loading for Database Pages
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-speed database loading processes face challenges in achieving parallel writing of records without locking memory pages, which hinders speed and leads to issues like tuple overwriting and reduced page usage rates due to hardware restrictions on simultaneous access to cache memory.
Innovation Solution
The solution involves using shared memory that allows simultaneous access by multiple processes, enabling parallel writing of tuples without locking pages, thereby improving load processing speed and preventing tuple overwriting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple processes write records in parallel to a page in cache memory, then load processing speed is improved, but tuple overwriting occurs and page usage rate decreases due to hardware restrictions on simultaneous access
Solution Approach 1:
The patent introduces shared memory as an intermediary between multiple writing processes and the page data. Instead of allowing direct parallel writes to the page in cache memory (which causes overwriting), the system uses shared memory as a buffer where multiple processes can write simultaneously without conflict, and then reads the complete data later. This mediator resolves the contradiction by enabling parallel processing while preventing data corruption.
2Reliability
If a page is locked to prevent simultaneous access, then data integrity is maintained, but load processing speed decreases due to serialization of writing operations
Solution Approach 1:
The patent segments the writing process into two distinct phases: a parallel writing phase where multiple processes write to shared memory simultaneously without locking, and a sequential reading phase where data is read from shared memory to the page. This segmentation allows the system to achieve both high concurrency during writing and data integrity during reading, resolving the contradiction between speed and reliability.
3Productivity
If simultaneous writing to shared memory is allowed, then parallel processing capability is improved, but writing control becomes more complex
Solution Approach 1:
The shared memory structure is designed to be inherently safe for parallel writes without requiring complex locking mechanisms or coordination protocols. Each writing process independently writes to its designated portion of the shared memory buffer, and the structure itself prevents overwriting through its design. This self-service approach enables parallel processing while keeping the control mechanism simple.
Data Source
AI summary
A non-transitory computer-readable recording medium having stored therein a program for causing a computer to execute a processing, the processing includes allocating a plurality of records to a page in shared memory that is able to be accessed simultaneously by a plurality of processings; receiving the plurality of records; writing, based on the plurality of records, information of writing region to the page for each of the plurality of records, and generating a writing processing corresponding to record for the plurality of records; generating, based on written the record to the writing region indicated by the information on the page by the writing processing executed, the page with at least of one of the record written; and loading the page generated to the database.