Sparse Distributed Representation for Semantic Similarity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems for clustering data documents primarily focus on predicting cluster positions or analyzing cluster structures, but they do not effectively utilize the resulting 'clustering map' as a 'distributed semantic projection map' for explicit semantic definition of data document constituents, limiting their ability to identify similarity between data items across different contexts.
Innovation Solution
A method is developed that generates sparse distributed representations (SDRs) for data items within a set of documents, allowing for the identification of similarity by clustering documents in a two-dimensional metric space, associating coordinate pairs, and determining occurrence information to create SDRs, which are then used to calculate distances between filtering criteria and streamed documents.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional clustering systems are used to organize data documents, then documents can be grouped by type and cluster structure analyzed, but the systems cannot effectively identify semantic similarity between data items across different contexts
Solution Approach 1:
The patent transforms documents from conventional clustering representations into sparse distributed representations (SDRs) with specific parameters (dimensionality, sparsity level, fill grade). This parameter transformation enables semantic similarity measurement by mapping documents into a unified vector space where cosine similarity can be computed, resolving the inability to identify semantic similarity across different document types and contexts.
Solution Approach 2:
The patent introduces SDRs as an intermediary representation between conventional document clustering and semantic similarity measurement. These SDRs serve as a mediator that translates diverse document types into a common format, enabling the similarity engine to compute distances and identify semantic relationships that would be impossible to detect using traditional clustering methods alone.
2Measurement precision
If detailed occurrence information is collected for each term to generate accurate SDRs, then semantic representation precision improves, but computational complexity and processing time increase
Solution Approach 1:
The patent extracts only the essential occurrence information needed for SDR generation (term frequency, document frequency, coordinate pairs) while discarding redundant textual details. This extraction approach maintains SDR representation accuracy by preserving the critical semantic features while reducing the data volume that requires processing, thereby lowering computational complexity without sacrificing precision.
Solution Approach 2:
The patent applies different processing qualities to different parts of the data: high-precision tracking of term occurrences and coordinate pairs for semantic accuracy, while using aggregated statistics (document frequencies, total occurrences) for the overall SDR construction. This local quality differentiation ensures accurate semantic representation while avoiding unnecessary computational overhead in less critical areas.
3Loss of information
If a comprehensive clustering map is generated for all data documents, then complete semantic coverage is achieved, but memory requirements and storage costs increase
Solution Approach 1:
The patent changes the parameter of representation from dense document-text mappings to sparse distributed vectors with controlled fill grades. By limiting the density of non-zero elements in SDRs while maintaining high dimensionality, the system achieves comprehensive semantic coverage through the distributed nature of the representations, while the sparsity constraint dramatically reduces memory requirements compared to storing complete clustering maps for all documents.
Solution Approach 2:
The patent creates compressed copies of document semantics in SDR form, which capture the essential semantic information in a compact representation. Instead of storing complete clustering maps and all term occurrences for every document, the system generates SDR copies that encode the same semantic content in a space-efficient format, maintaining information coverage while reducing storage requirements.
Data Source
AI summary
A method enables identification of a similarity level between a user-provided data item and a data item within a set of data documents. The method includes a representation generator determining, for each term in an enumeration of terms, occurrence information. The representation generator generates, for each term, a sparse distributed representation (SDR) using the occurrence information. The method includes receiving, by a filtering module, a filtering criterion. The method includes generating, by the representation generator, for the filtering criterion, at least one SDR. The method includes generating, by the representation generator, for a first of a plurality of streamed documents received from a data source, a compound SDR. The method includes determining, by a similarity engine executing on the second computing device, a distance between the filtering criterion SDR and the generated compound SDR. The method includes acting on the first streamed document, based upon the determined distance.