Bitmap Association Sets for Concurrent Entity Linking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing solutions for managing large association sets in enterprise content management are inefficient, leading to high storage costs, long query times, and concurrency issues due to the need for extensive locking during association and dissociation operations.
Innovation Solution
The approach involves assigning unique identifiers to entities and representing association sets as bitmaps over an integer space, partitioning these sets into segments for efficient storage and querying, allowing for concurrent operations without locking entire sets, and implementing automatic filtering and deferred merge for deleted entities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional database tables with links are used to represent associations between entities, then transactional guarantees can be provided, but the system becomes unusable for other operations during creation or deletion operations due to extensive locking
Solution Approach 1:
The patent segments the association set into fixed-size blocks, allowing operations to be performed on individual blocks rather than the entire association set. This enables concurrent operations on different blocks, maintaining system availability while providing transactional guarantees within each block.
Solution Approach 2:
The patent performs preliminary actions by pre-allocating blocks and preparing transaction logs before actual association operations. This allows the system to maintain consistency and provide transactional guarantees without blocking the entire system, as operations are prepared and executed in a controlled manner on specific blocks.
2Quantity of substance
If traditional database tables are used to store large association sets, then complete associations can be tracked, but storage costs become very high and query times become very long
Solution Approach 1:
The patent divides the association set into fixed-size blocks that can be independently stored and queried. This segmentation allows the system to handle large association sets efficiently by loading only relevant blocks into memory during queries, significantly reducing query response time while maintaining complete association tracking.
Solution Approach 2:
The patent introduces a block dimension to the traditional flat table structure, organizing associations hierarchically into blocks and individual entries. This dimensional change enables efficient indexing and retrieval operations, reducing query time from O(n) to O(1) for block-level access while maintaining complete association data.
3Quantity of substance
If traditional database tables are used to store large association sets, then all associations can be persisted, but storage space requirements become very large
Solution Approach 1:
The patent segments the association set into fixed-size blocks, allowing efficient compression and selective storage. Each block can be independently compressed and stored, reducing overall storage requirements while maintaining complete association data. The block structure also enables efficient deduplication across the entire association set.
Solution Approach 2:
The patent changes the storage parameters by using fixed-size blocks with predetermined block sizes instead of variable-size rows. This parameter change enables more efficient storage allocation, compression, and memory management, significantly reducing storage space requirements while maintaining complete association data persistence.
4Reliability
If transactions are used to create associations with locking, then data consistency is maintained, but the system becomes unusable for concurrent operations
Solution Approach 1:
The patent segments the association set into blocks that can be independently locked and operated on. This allows multiple concurrent transactions to operate on different blocks simultaneously, maintaining data consistency within each block while enabling system-wide concurrent operations that would otherwise be blocked in a traditional table-based approach.
Data Source
AI summary
Processing a database query for sets of data includes assigning a unique identifier from an integer space to each entity within data and creating one or more sets of entities each pertaining to a corresponding entity within the data. A representation is then generated on disk for each set of entities, wherein each representation encompasses and is suited for a range of the unique identifiers of entities within a corresponding set and indicates a presence of an entity within that corresponding set. Finally, a query is processed based on the representation for each set of entities to retrieve data satisfying the query, wherein the representation provides a constant time for association and dissociation operations that are append-only operations with deferred merge and automatic filtering of deleted and duplicate entities at query time.