NoSQL Inverted Index Segmentation for Semi-Structured Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional NoSQL database index systems face low query efficiency and low update efficiency, particularly when dealing with large-scale semi-structured data, leading to unacceptable throughput and inability to process TB-sized documents effectively.
Innovation Solution
The method involves preprocessing semi-structured data to create a data table with a combined primary key including a structure thread and sequence value, and an inverted index table with a combined primary key of data source, structure thread, and keyword, allowing for efficient querying and updating by parsing only relevant sequence values, thereby improving query efficiency and reducing the need to read the original inverted index table during updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a conventional inverted index table is used to store keyword-document mappings in NoSQL database, then the database can handle unstructured and semi-structured data, but the query efficiency becomes low when dealing with large-scale data because the entire inverted index table needs to be searched
Solution Approach 1:
The patent segments the inverted index table into multiple partitioned tables based on primary key ranges. Each partitioned table stores a specific range of document IDs, allowing the query process to search only relevant partitions rather than the entire index table. This segmentation directly reduces query time while maintaining the capability to handle large-scale unstructured and semi-structured data.
2Reliability
If the entire inverted index table is searched to find keyword locations, then all keyword occurrences can be found, but the search time increases exponentially with the amount of data
Solution Approach 1:
The inverted index is divided into multiple partitioned tables where each table covers a specific primary key range. During keyword search, only the partitioned tables containing the relevant primary key range are queried, ensuring that all keyword occurrences within that range are found while significantly reducing the total search time compared to scanning the entire index table.
Solution Approach 2:
Instead of searching the entire inverted index table, the system performs partial action by searching only the specific partitioned tables that contain the relevant data range. This partial search approach maintains completeness for the queried range while avoiding the excessive time cost of searching unnecessary portions of the index.
3Stability of the object's composition
If the original inverted index table is read during index updates, then existing keyword mappings can be preserved, but the updating speed becomes unacceptable for very large databases
Solution Approach 1:
The patent segments the index update operation into partitioned table level operations. When updating the inverted index, only the specific partitioned tables that need updating are modified, rather than reading and processing the entire inverted index table. This segmentation enables parallel processing and significantly improves index updating speed while preserving the stability of the overall index structure through consistent partition management.
Solution Approach 2:
The system performs preliminary actions by pre-defining the partition structure and primary key ranges before index updates. This allows update operations to directly target specific partitions without needing to read or process the entire index table, thereby improving updating speed while maintaining structural stability through predetermined partition boundaries.
4Device complexity
If a single primary key is used in the inverted index table, then the structure is simple, but it cannot efficiently support partitioned querying and updating operations
Solution Approach 1:
The patent introduces a composite primary key structure that combines the original primary key with partition range information. This segmented key structure enables efficient routing of queries and updates to specific partitioned tables while maintaining a relatively simple overall structure. The composite key approach improves query and update throughput by enabling direct partition location without sacrificing excessive structural simplicity.
Data Source
AI summary
Semi-structured source data is preprocessed to obtain text partitions to be stored into a data table with a first combined primary key including a structure thread primary key and a sequence value primary key. The structure thread primary key identifies a structure thread that is segmented into several consecutive intervals according to a determined or predetermined sequence. An inverted index table, created for the preprocessed text partitions, includes a second combined primary key including the structure thread primary key and a keyword primary key. Corresponding to values of the primary keys in the second combined primary key, related text partition sequence IDs are recorded as index values of the inverted index table. Index values having a same keyword primary key value but different structure thread primary key values are located in different rows in the inverted index table. The present techniques improve query efficiency of database index and facilitate updating.