Knowledge Base Entity Classification Through Parallel Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently classifying large knowledge bases due to their massive size, which leads to high computational intensity and storage requirements, resulting in slow query execution times and data manipulation difficulties.
Innovation Solution
Partitioning the knowledge base into smaller subsets and classifying entities in parallel using a set of subclasses, with methods such as JSON Lines format and streaming APIs to manage updates, reducing computational load and improving query efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the knowledge base is processed as a single large dataset, then classification completeness is maintained, but query execution time increases significantly and computational resources are overwhelmed
Solution Approach 1:
The knowledge base is divided into multiple partitions, each containing a subset of entities. Each partition is processed independently and in parallel, reducing the computational burden on individual processing units and enabling faster query execution while maintaining overall classification completeness across all partitions
2Reliability
If the entire knowledge base is loaded into memory for processing, then classification accuracy is maintained, but storage requirements and memory consumption increase significantly
Solution Approach 1:
The knowledge base is segmented into multiple partitions that can be stored and processed separately. This allows the system to load only the necessary partition into memory at any given time, significantly reducing storage requirements and memory consumption while maintaining classification accuracy through consistent processing of each partition
Solution Approach 2:
A copy of the knowledge base is obtained and partitioned for processing. This allows the original knowledge base to remain intact while working with smaller, manageable copies that can be loaded into memory efficiently, reducing storage requirements during the classification process
3Speed
If the knowledge base is processed sequentially, then data consistency is maintained, but processing speed decreases significantly
Solution Approach 1:
The knowledge base is divided into independent partitions that can be processed in parallel. Each partition maintains its own data consistency through independent processing, while the overall system achieves faster processing speeds by concurrently handling multiple partitions without requiring sequential processing
Solution Approach 2:
The knowledge base is pre-partitioned into independent subsets before processing begins. This preliminary organization allows subsequent processing to proceed in parallel while maintaining data consistency within each partition, as the partitioning structure is established beforehand to support concurrent operations
Data Source
AI summary
Disclosed herein are system, apparatus, device, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for efficient entity classification. In an embodiment, a set of subclasses associated with a particular entity class is obtained via a query service of a knowledge base. A copy of the knowledge base is obtained, the copy of the knowledge base specifying a plurality of entities. The copy of the knowledge base is partitioned into a plurality of partitions, each partition of the plurality of partitions specifying a respective subset of the plurality of entities. For each partition of the plurality of partitions, at least one entity in the respective subset of the plurality of entities is classified with the particular entity class based on the one or more subclasses of the set of subclasses.