Bitset Entity Indexing for Catalog Microservice Communication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In content provider applications, especially in streaming platforms, transmitting and storing metadata associated with a content catalog is inefficient due to the need to handle large sets of entity identifiers, leading to computationally expensive operations and significant scalability challenges.
Innovation Solution
The method involves loading an index of entity identifiers into memory at a microservice, composing a bitset to represent sets of entities, and transmitting this bitset to other microservices, allowing for efficient decoding and reducing message size and computational costs by using bitwise operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If entire entity identifier strings are transmitted in JSON format, then complete metadata information is communicated, but message size becomes inordinately large
Solution Approach 1:
The patent maintains local copies of the entity index at each microservice. Instead of transmitting full entity identifier strings, services transmit compact ordinal values that reference the locally cached index, dramatically reducing message size while preserving complete metadata information through the mapping relationship.
Solution Approach 2:
The patent transforms entity identifiers from human-readable strings into compact ordinal numbers through indexing. This parameter transformation reduces the size of transmitted data from potentially long strings to small integer values, while the local index enables reconstruction of the original information when needed.
2Measurement precision
If entire entity identifier strings are transmitted, then accurate entity identification is achieved, but transmission time increases significantly
Solution Approach 1:
By caching the entity index locally at each microservice, the system enables rapid lookup and transmission of compact ordinal references instead of full identifier strings. This copying approach maintains precise entity identification through the mapping relationship while dramatically reducing transmission time.
3Adaptability or versatility
If string-based metadata is used for catalog entities, then flexible data representation is maintained, but computational operations become expensive
Solution Approach 1:
The patent replaces string-based metadata operations with numeric ordinal-based operations. Bitset operations on ordinal values are computationally much more efficient than string manipulations, while the local index maintains the flexibility to represent diverse catalog entities. This substitution transforms expensive string processing into efficient bitwise operations.
4Loss of information
If full metadata strings are stored in memory, then complete entity information is available, but memory consumption increases
Solution Approach 1:
The patent transforms stored metadata from full entity identifier strings into compact ordinal values in the index. This parameter transformation dramatically reduces memory consumption while preserving complete entity information through the mapping relationship. The local index stores only the essential mapping from ordinals to identifiers, enabling efficient storage and retrieval.
Data Source
AI summary
In various embodiments a computer-implemented method for communicating sets of entities in a content catalog is disclosed. The method includes loading an entity index into memory at a microservice, where the entity index comprises entity identifiers corresponding to titles in a catalog, and where each identifier in the entity index is mapped to an ordinal number. The method also includes composing a message including a bitset to identify titles from the catalog, where a bit in the bitset is set if a position of the bit in the bitset corresponds to a respective ordinal number in the entity index associated with the one or more titles. Additionally, the method includes transmitting the message to a different microservice where a memory for the recipient microservice comprises a copy of the entity index, and where the bitset comprised within the message is decoded into entity identifiers using the entity index.