Index Compressed Flat Files for Data Cube Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage techniques for data cubes, such as relational databases, face inefficiencies in accessing and retrieving data due to locking mechanisms, storage space usage, and processing power requirements, especially in high-speed network communications.
Innovation Solution
Storing data cubes in index compressed flat files (ICFFs) allows for efficient data access by grouping records based on primary keys and hierarchies, enabling direct reading and compression, which minimizes processing power and storage space usage without the need for locking techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If relational databases are used to store data cubes, then data can be stored with structured relationships and indexing, but data access requires locking mechanisms that reduce access speed and increase processing overhead
Solution Approach 1:
The patent segments the data cube into multiple chunks or partitions that can be independently stored and accessed. Each chunk contains a subset of the data records, allowing parallel access without locking conflicts. This segmentation enables multiple users or processes to access different chunks simultaneously, eliminating the need for centralized locking mechanisms while maintaining data integrity.
Solution Approach 2:
The patent extracts the locking mechanism entirely from the data storage and access system by using immutable flat files. Instead of using traditional database tables that require locks for concurrent access, the data is stored in a read-only format where new data is appended as new files or chunks. This extraction of the locking function eliminates the associated complexity and speed penalties while maintaining data consistency through the append-only nature of the storage.
2Quantity of substance
If traditional database storage is used, then data can be organized with relationships and indexes, but storage space efficiency is reduced due to overhead from locking mechanisms and processing structures
Solution Approach 1:
The patent removes unnecessary processing structures and overhead from the storage system by using simple flat files instead of complex database tables. The data is stored in a streamlined format without the need for locking mechanisms, transaction logs, or complex indexing structures. This extraction of overhead elements significantly improves storage space efficiency while reducing the computational resources required for data management.
Solution Approach 2:
The patent changes the fundamental parameters of data storage by transitioning from mutable database tables to immutable flat files. This parameter change eliminates the need for version control, locking, and complex update mechanisms, thereby reducing storage overhead. The immutable nature of the files allows for more efficient compression and reduces the space required for maintaining data consistency across multiple users and processes.
3Productivity
If data is stored in flat files without structured relationships, then storage space is optimized and direct reading is enabled, but data retrieval efficiency is reduced without indexing structures
Solution Approach 1:
The patent segments the data cube into logical chunks that can be independently accessed and processed. Each chunk is self-contained and can be read directly without requiring global indexing structures. This segmentation enables efficient data retrieval by allowing the system to access only the relevant chunks based on query criteria, improving productivity while avoiding the overhead of maintaining complex global indexes across the entire data cube.
Solution Approach 2:
The patent performs preliminary organization of data into chunks during the data loading phase, arranging records within each chunk in a consistent order. This preliminary action eliminates the need for complex runtime indexing and searching, as the data is already organized for efficient access. The preliminary organization allows for direct reading and fast retrieval while maintaining simplicity in the storage structure.
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
Among other things, we describe a technique for storing data of a data cube in one or more flat files. We also describe a technique for processing a query to access data of a data cube. These techniques can be implemented in a number of ways, including as a method, system, and/or computer program product stored on a computer readable storage device. One of the techniques includes receiving a set of data records having at least two dimensions, generating a set of grouped data records ordered by cardinality, and generating and storing at least one flat file containing the set of grouped data records, wherein a particular data record of the grouped data records includes a primary key that can be used to identify data of the particular data record in response to a request.