File Storage Indexing for Compressed Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face challenges in efficiently managing and storing individually accessible data units, particularly in handling file updates and compression, which can lead to storage inefficiencies and errors.
Innovation Solution
A method that determines the length of a file and updates a data structure when necessary, simultaneously adds data to a buffer and transfers it to a compressed file, and manages status information to coordinate the creation and deletion of files, ensuring efficient storage and retrieval of records while checking for errors and maintaining data integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is compressed when stored, then storage requirements are reduced, but decompression is required when accessed increasing processing time
Solution Approach 1:
The patent pre-calculates and stores decompression offset values in an index structure before data is actually accessed. This preliminary action allows the system to quickly locate and decompress only the necessary portions of compressed data without having to process the entire compressed file, thereby reducing access time while maintaining compression benefits.
Solution Approach 2:
The patent divides compressed data into separate blocks or segments, each with its own decompression parameters stored in an index. This segmentation allows selective decompression of individual blocks rather than the entire compressed dataset, reducing the time required to access specific data while preserving overall storage efficiency.
2Productivity
If file length is tracked in memory locations, then data structure updates can be efficiently managed, but additional memory requirements are incurred
Solution Approach 1:
The patent uses the existing index structure to serve multiple functions: it stores both the decompression parameters and the file length information. By making the index multi-functional, the system avoids allocating separate memory structures for length tracking, thus improving data structure update efficiency without significantly increasing memory requirements.
3Productivity
If data is simultaneously added to files and buffers, then data processing throughput is improved, but coordination complexity increases
Solution Approach 1:
The patent introduces an intermediary index structure that mediates between the buffer and the compressed file. This index acts as a coordination layer that tracks data additions to both the buffer and file, managing the complexity of simultaneous operations without requiring complex direct coordination between all components.
Solution Approach 2:
The system implements feedback mechanisms where the index continuously monitors and updates the state of both the buffer and compressed file. This feedback loop allows the system to maintain consistency and coordinate operations automatically, reducing the apparent complexity by making coordination explicit and automated rather than requiring complex control logic.
4Reliability
If status information is locked during file operations, then data integrity is maintained, but access speed is reduced
Solution Approach 1:
The patent segments the status information into multiple locks corresponding to different files or data structures. This allows fine-grained locking where only the specific portion being modified is locked, while other portions remain accessible. This maintains data integrity for the modified portion while preserving access speed for other data.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
A method includes determining a length of a file (702) and storing the length of the file in a first memory location (704). An endpoint of a last complete record within the file is determined (706) and the endpoint is stored in a second memory location (708). The length of the file stored in the first memory location is compared to a current length of the file (710), and a data structure associated with the file is updated beginning at the endpoint if the current length of the file exceeds the length of the file stored in the first memory location (712).