In-Memory Stream Vector for Distributed Social Network Data Merging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed social networks, efficiently computing a time-ordered stream that combines substreams for individual users and tracking changes since the last view is challenging due to data being distributed across multiple servers, making querying and updating operations complex.
Innovation Solution
The implementation of in-memory data structures and algorithms that allow for efficient querying and updating of time-ordered streams and substreams without locking, enabling the generation of globally sorted streams, intersection of queries, and tracking of notifications, using data structures like the Stream Vector and Substream Start Vector, and algorithms that maintain stream integrity across multiple readers and servers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is distributed across multiple servers, then scalability and system capacity are improved, but querying and stream generation operations become more complex and slower
Solution Approach 1:
The system divides the distributed data into substreams organized by user, with each substream maintained separately. The Stream Vector data structure segments stream data into manageable units that can be independently processed, allowing efficient merging without requiring complex coordination across all distributed servers for every operation.
Solution Approach 2:
The patent introduces intermediary data structures (Stream Vector, Substream Start Vector, and Merge State) that act as mediators between the distributed servers and the final merged stream. These intermediaries buffer and organize data locally, reducing the complexity of direct server-to-server querying and enabling faster stream generation.
2Speed
If traditional distributed querying methods are used, then data consistency across servers is maintained, but operation speed and user experience responsiveness deteriorate
Solution Approach 1:
The system performs preliminary organization of stream data into user-specific substreams and maintains Substream Start Vectors that pre-compute and store starting positions for each user's data. This preliminary structuring allows rapid access and merging operations without requiring complex real-time coordination, thus improving speed while maintaining integrity through pre-established data organization.
Solution Approach 2:
The patent replaces traditional mechanical distributed locking and coordination mechanisms with in-memory data structures and algorithms. By using Stream Vectors and Merge States that can be efficiently manipulated in memory, the system achieves faster operations while maintaining stream integrity through algorithmic guarantees rather than mechanical locking protocols.
3Productivity
If in-memory data structures are used for fast access, then operation speed is improved, but memory consumption and system resource requirements increase
Solution Approach 1:
The Stream Vector data structure segments stream data into fixed-size blocks or chunks, allowing the system to load and process only relevant portions into memory at any given time. This segmentation enables efficient query processing with controlled memory usage, as the system can work with discrete units rather than requiring entire streams in memory simultaneously.
Solution Approach 2:
The system uses Merge State to track only the necessary portion of stream data that needs to be merged at any given moment, rather than maintaining complete stream states for all users. This partial action approach processes only the relevant subset of data in memory, improving query efficiency while limiting memory consumption to what is strictly necessary for current operations.
Data Source
AI summary
Various embodiments provide data structures and algorithms for efficiently computing a time ordered stream that is the combination of a time ordered substream for individual persons of interest. In addition, various embodiments provide an approach for identifying how many changes of interest have happened to a particular stream since it was last viewed.