Indexing Semi-Structured Documents via Key-Value Store
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional databases and search engines are ineffective in processing semi-structured data that does not conform to a rigid schema, leading to poor quality or inefficient search results due to the lack of exploitation of the structure present in such data.
Innovation Solution
A computer-implemented method and system that uses a key-value store to create multiple indexes (document index, field index, and columnar index) based on semi-structured documents, allowing for efficient querying by converting fields into key-value pairs and generating key-expressions to match search criteria.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional databases are used to store semi-structured data, then data storage is possible, but search quality and efficiency deteriorate because the data does not conform to rigid schemas
Solution Approach 1:
The patent segments the indexing process into multiple specialized indexes (document index, field index, columnar index) that handle different aspects of semi-structured data separately. Each index type is optimized for specific query patterns, allowing the system to maintain high search quality while accommodating flexible schemas without requiring a rigid overall structure.
Solution Approach 2:
The system changes the parameter of data representation by converting semi-structured data into a normalized key-value format with explicit field names, values, and types. This transformation allows the data to be stored flexibly while enabling efficient querying through generated key-expressions that match search criteria.
2Productivity
If conventional search engines are used to search semi-structured data, then searching capability is provided, but search efficiency deteriorates because the structure of semi-structured data is not exploited
Solution Approach 1:
The system performs preliminary action by pre-processing semi-structured data during ingestion and converting it into a normalized key-value format with explicit field metadata. This preliminary transformation enables efficient search operations later by allowing direct key-expression matching without needing to parse complex semi-structured formats during query execution.
Solution Approach 2:
The patent introduces an intermediary indexing layer that sits between the semi-structured data storage and the search engine. This intermediary converts semi-structured data into a normalized key-value representation and creates multiple specialized indexes, acting as a mediator that enables efficient searching without requiring changes to the underlying storage or query interfaces.
3Reliability
If multiple indexes are created to improve search efficiency, then search quality improves, but system complexity increases
Solution Approach 1:
The patent applies universality by designing a unified key-value store infrastructure that serves multiple indexing functions simultaneously. The same key-value store backend supports document indexes, field indexes, and columnar indexes, allowing the system to maintain multiple specialized indexes without proportionally increasing overall system complexity.
Data Source
AI summary
A search system processes queries for accessing information stored in documents. A document comprises fields. The search system stores a plurality of indexes in a key-value store. Each index comprises key-value pairs. A key of a key-value pair is obtained by combining field data describing a field of a document. The value of each field is stored as an individual key-value in the key-value store. The search system receives a query requesting information stored in documents and specifying a search criteria. The search system builds a key-expression based on the search criteria and uses one or more indexes to find key-value pairs matching the key-expression. The search system finds the requested information based on the matching key-value pairs and provides the requested information to the query source.