Lexical Text Storage Layout for Faster Compressed Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Efficient storage and retrieval of large datasets, particularly textual data, is resource-intensive and time-consuming in cloud computing systems, especially for non-relational data structures like columnar data structures.
Innovation Solution
A dual data structure approach is employed, comprising a primary file system for efficient compression and search using lexical ordering and delta compression, and a secondary data structure for managing updates and modifications, minimizing memory inefficiencies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in cloud computing systems, then storage capacity is increased, but storage space becomes scarce and retrieval becomes time-consuming
Solution Approach 1:
The patent divides the data storage system into multiple data structures organized in a hierarchical manner. Data is segmented into different levels of data structures (e.g., data blocks, data structures, data structure trees) that can be independently managed and accessed. This segmentation allows efficient retrieval by enabling the system to navigate to specific segments without scanning the entire dataset, thus reducing retrieval time while maintaining large storage capacity.
Solution Approach 2:
The patent introduces a multi-dimensional organizational structure for data storage, using hierarchical data structure trees with multiple levels and dimensions. This dimensional organization allows data to be accessed through multiple pathways and indices, enabling faster retrieval by selecting the most efficient access path based on the query type, thereby reducing retrieval time without compromising storage capacity.
2Quantity of substance
If large datasets are stored, then data volume is increased, but searching and retrieving desired data becomes resource-intensive
Solution Approach 1:
The patent implements pre-computed indices, metadata, and data structure trees that are built in advance during data ingestion and update operations. These preliminary structures enable rapid data location and retrieval without requiring intensive computational resources during query execution. The system performs sorting, indexing, and structural organization beforehand, so that when data needs to be retrieved, the computational burden is significantly reduced.
Solution Approach 2:
The patent introduces intermediate data structures such as indices, metadata layers, and data structure trees that act as mediators between the raw large dataset and the query operation. These intermediaries pre-process and organize data in a way that facilitates efficient searching, allowing the system to locate desired data with minimal computational resources by navigating through the intermediate structures rather than scanning the entire large dataset.
3Adaptability or versatility
If non-relational data structures are used, then data flexibility is improved, but storage efficiency and retrieval performance deteriorate
Solution Approach 1:
The patent applies different organizational strategies and data structure types to different portions of the data based on local characteristics and access patterns. Rather than using a uniform structure for all data, the system adapts the data structure organization to local data properties, allowing efficient storage and retrieval for each specific data type or access pattern while maintaining overall flexibility for diverse data formats.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
A method of and system of efficient storage of data entries containing textual data is disclosed. The method may include accessing a plurality of data entries in a dataset, arranging the plurality of data entries in the dataset in a lexical order, placing a predetermined number of the plurality of data entries in each of a plurality of subblocks, performing data compression on the plurality of data entries in each of the plurality of subblocks to reduce redundancy in the plurality of data entries and create compressed data entries, placing one or more subblocks in each of a plurality of page blocks, and storing each of the plurality of page blocks in a storage device to provide efficient searching and improved functionality for the dataset.