Database Table Segmentation for Large Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems face challenges in efficiently processing large amounts of data, particularly when data exceeds the addressable memory space of local memory, leading to inefficiencies and limitations in data manipulation and storage.
Innovation Solution
The method involves generating a database table in a peripheral storage device, allowing an application program to access and process data using database techniques, rather than relying solely on local memory, and includes locking and unlocking the database table for exclusive access, with the option to delete the table upon termination.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in local memory for processing, then processing speed is improved, but memory capacity is limited and data loss occurs upon power disruption
Solution Approach 1:
The patent divides the data storage and processing system into two segments: local memory for active processing and peripheral storage for persistent data retention. This segmentation allows the system to leverage the speed of local memory while overcoming its capacity and volatility limitations through peripheral storage.
Solution Approach 2:
The patent introduces a database table as an intermediary between local memory and peripheral storage. This database table serves as a buffer that allows the application program to access data using database techniques while storing data persistently in peripheral storage, resolving the contradiction between processing speed and storage capacity/durability.
2Quantity of substance
If data exceeds addressable memory space of local memory, then storage capacity is improved, but processing efficiency deteriorates
Solution Approach 1:
The database table acts as an intermediary layer that enables efficient access to large datasets stored in peripheral storage. By providing database accessing techniques (such as indexing, querying, and structured access), the system can process large amounts of data without being constrained by local memory addressable space while maintaining processing efficiency.
Solution Approach 2:
The patent transitions from a single-dimension storage approach (local memory only) to a two-dimension approach by adding peripheral storage with database table access. This dimensional change allows the system to access large datasets efficiently through database techniques while keeping frequently accessed data in local memory.
3Reliability
If database table is locked for exclusive access, then data integrity is improved, but access flexibility deteriorates
Solution Approach 1:
The patent implements dynamic locking where the database table is locked only during critical processing operations and unlocked when not in use. This dynamic approach ensures data integrity during processing while maintaining access flexibility when the application program is not actively processing data, resolving the contradiction between reliability and ease of operation.
Data Source
AI summary
Among other disclosed subject matter, a computer-implemented method for handling large amounts of data is to be initiated. The method includes receiving, using an application program executed from local memory of a computer system, portions of information from a first file. The method includes generating a database table in a peripheral storage device. The method includes storing the portions of information in the database table of the peripheral storage device. The method includes locking the database table, wherein the locking exclusively provides the application program a database accessing technique to the database table. The method includes processing by the application program the portions of information in the database table using the database accessing technique. The method includes clearing the database table upon a termination of the application program. The method includes, unlocking the database table upon the termination of the application program.