Structured Record Retrieval via Native Format Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for storing and retrieving structured records, such as EDIFACT messages, often require conversion to a tabular form, leading to increased storage space and potential inconsistencies, especially when dealing with many optional fields.
Innovation Solution
A method that indexes structured records directly in their native format using a grammar-based approach, parsing field values to create an index structure with bit vectors, allowing for efficient retrieval without the need for tabular representation, thereby reducing storage requirements and minimizing inconsistencies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If records are converted to tabular form for storage, then retrieval efficiency is improved, but storage space requirement increases
Solution Approach 1:
The patent segments the record data into mandatory fields and optional fields, creating separate storage paths for each. Mandatory fields are stored in tabular form for efficient querying, while optional fields remain in their native hierarchical format. This segmentation allows the system to achieve fast retrieval on commonly queried mandatory fields without converting and storing all optional fields in tabular form, thus balancing retrieval efficiency with storage space utilization.
Solution Approach 2:
The patent applies different storage qualities to different parts of the record structure. Mandatory fields that are frequently queried receive the benefit of tabular storage with indexes for fast retrieval, while optional fields that are less frequently accessed remain in compact native format. This local differentiation of storage quality optimizes the trade-off between retrieval efficiency and storage space by applying the more space-consuming tabular format only where it provides the most value.
2Ease of operation
If records are converted to tabular form, then query processing is simplified, but data consistency becomes harder to maintain
Solution Approach 1:
The patent introduces an intermediary layer (the hybrid storage system with mandatory field tables and optional field native format) that mediates between the query processing requirements and the data consistency requirements. This intermediary structure allows queries to be processed efficiently through the tabular mandatory fields while maintaining a single source of truth in the native format, reducing consistency issues that would arise from maintaining separate converted copies.
Solution Approach 2:
The patent creates a selective copy of only the mandatory fields in tabular form, rather than copying all fields. This partial copying approach simplifies query processing for the most important fields while minimizing the risk of consistency errors, since there is less data to synchronize between formats. The optional fields remain in their original location, eliminating the need to maintain consistency between multiple storage representations.
3Productivity
If all fields are stored in tabular form, then retrieval of any field is efficient, but storage space doubles
Solution Approach 1:
The patent segments fields into mandatory and optional categories, storing them in different formats. This segmentation enables the system to achieve efficient retrieval for mandatory fields through tabular storage while avoiding the space overhead of converting optional fields, thus preventing storage space from doubling while maintaining productivity for critical data access.
Solution Approach 2:
The patent applies partial action by converting only the necessary mandatory fields to tabular form rather than all fields. This partial conversion achieves sufficient retrieval efficiency for the most important data while avoiding the excessive storage cost of converting every field, thus resolving the contradiction between retrieval speed and storage space.
Data Source
AI summary
An approach to structured record retrieval permits transmission and storage of records in a native concise format, without requiring that the records be interpreted and stored in a tabular form. Such storage of the records in a tabular form might double the space required, and more generally, requires substantially more space in applications in which there are many optional elements. In some embodiments, each message is parsed according to a specification of the message structure (e.g., according to a “grammar” for the message), and during parsing field values in predefined positions in the structure are extracted and added to an index structure that associates record identifiers with the (position, value) pairs.