Inverted Indexing System File Segmentation for I/O Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for establishing an inverted indexing system are inefficient, particularly when dealing with large amounts of data, as they require multiple I/O disk accesses and external storage device usage, leading to low efficiency and slow data processing.
Innovation Solution
The method involves writing inverted indexes into an inverted file and separate description files, allowing for the establishment of an inverted indexing system without modifying the file header, thereby eliminating the need for external storage and enhancing efficiency by partitioning data records and combining files based on functional mapping relations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the conventional method is used to establish the inverted indexing system by writing inverted indexes to an inverted file and then returning to the file header to fulfill description information, then the inverted indexing system can be established, but the process requires multiple I/O disk accesses and external storage device usage, resulting in low efficiency and slow data processing
Solution Approach 1:
The patent separates the inverted file and description file into two independent files. The inverted file stores only the inverted index data, while the description file stores metadata such as file size and record count. This segmentation eliminates the need to repeatedly access and modify the file header, as description information is written once to a separate file after inverted index writing is complete, significantly reducing I/O operations and establishing time for large datasets
Solution Approach 2:
The patent performs preliminary writing of all inverted indexes to the inverted file before writing the description information to the description file. This ensures that the inverted file is completely populated with index data before its metadata is finalized, eliminating the need for subsequent modifications to the file header and allowing the system to proceed directly to data processing without additional I/O overhead
2Reliability
If the inverted file is written in sequence to a local storage device to form an inverted file containing description information, then the file structure is maintained, but multiple I/O disk accesses are required, causing low efficiency
Solution Approach 1:
The patent divides the original single inverted file into two separate files: the inverted file containing only index data and the description file containing metadata. This segmentation maintains data integrity by clearly separating data storage from metadata storage, while simultaneously improving productivity by eliminating repeated I/O operations to modify file headers during the indexing process
Solution Approach 2:
The patent extracts the description information (metadata) from the inverted file structure and stores it in a separate description file. This extraction removes the need to repeatedly access and modify the inverted file's header section, allowing the inverted file to be written once in sequence without subsequent modifications, thereby maintaining structural integrity while dramatically improving establishment speed
3Ease of manufacture
If external storage device is used to store the inverted file before uploading to the improved system, then the inverted indexing system can be established, but additional processes are required, resulting in low efficiency
Solution Approach 1:
The patent enables the computing device to directly write both the inverted file and description file to the distributed file system without requiring external storage devices. The system serves itself by performing all file operations locally, eliminating the intermediate step of using external storage and subsequent uploading processes, thereby simplifying the workflow and improving establishment efficiency
Data Source
AI summary
The present disclosure relate to techniques for establishing an inverted indexing system and related data processing. The techniques may include writing, by a computing device, inverted indexes of a massive amount of data records into at least one inverted file. The computing device may then write description information of the written inverted file into a description file associated with the inverted file, and establish the inverted indexing system based on the inverted file and the description file of the inverted file. The techniques enhance efficiency in establishing the inverted indexing system and in processing data using the systems.