Hash Indexing for Unstructured Data in Structured Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for storing unstructured data in structured databases are inefficient, as they require sequential reading of large data sets to search for specific documents, and do not allow for full utilization of advanced access methods and data management techniques available in structured databases.
Innovation Solution
The method involves receiving an unstructured data record, parsing it to determine its characterization, obtaining a key pair with a primary key-name and key-value, hashing the key-value to obtain a hash value, and updating the data record to include the hash value as an indexing key. This modified data record is then stored in a structured database, and the index is updated to include an entry for the modified data record with the hash value.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If unstructured data is stored as a continuous stream of bytes in a non-relational database, then rapid deployment and flexible data definition are achieved, but search operations require sequential reading of large data sets which is time and resource intensive
Solution Approach 1:
The patent segments unstructured data into structured records with defined fields and data types. Each record is parsed into discrete components (headers, fields, values) that can be independently indexed and accessed, eliminating the need to sequentially read entire data streams during search operations.
Solution Approach 2:
The patent performs preliminary structuring and indexing of unstructured data during the storage phase. By organizing data into a structured format with defined schemas and creating indexes before search operations occur, the system prepares data for rapid retrieval without requiring sequential scanning during actual search operations.
2Quantity of substance
If unstructured data is stored as a continuous data stream in a structured database, then storage is achieved, but indexing of the data is not allowed
Solution Approach 1:
The patent changes the structural parameters of unstructured data by imposing a defined schema with specific data types, field lengths, and record formats. This transformation enables the data to be indexed by structured database mechanisms while maintaining the ability to store large volumes of information.
3Ease of operation
If additional indexes are created on top of or separate from the primary index to store unstructured data, then data retrieval is enabled, but additional overhead is required when storing data and unnecessary delays occur when searching for specific documents
Solution Approach 1:
The patent creates a universal structured data format that serves multiple functions simultaneously: it enables efficient storage, supports primary indexing, and allows for secondary indexes to be created on specific fields. This multi-functional approach eliminates the need for separate indexing mechanisms and reduces overall system complexity.
Data Source
AI summary
In one embodiment, a method includes receiving an unstructured data record and parsing the unstructured data record to determine a characterization of the unstructured data record. The characterization includes determination of: a number of fields, a length for the fields, and an order of the fields as stored in the unstructured data record. The method also includes obtaining a first key pair that includes a primary key-name associated with a first key-value and hashing at least the first key-value to obtain a hash value. In addition, the method includes updating the unstructured data record to include the hash value as an indexing key thereby creating a modified data record. Also, the method includes storing the modified data record in a structured database. Moreover, the method includes updating an index of the structured database to include an entry for the modified data record, the entry including the hash value.