Data Buffering Layer Routing for Long Relation Chain Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems for micro-blog users with long relation chains face slow data operation speeds due to inefficient processing by the stream and friend modules, leading to a low speed in processing follower information.
Innovation Solution
A data processing method and system that involves a data buffering layer pushing operation requests to a service layer, which caches requests by a minimum storage unit divided based on the user with a long relation chain, and performs operations on a specified database when preset conditions are met, allowing for batch operations and reduced data processing burden.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If one database stores both operation requests and entire follower information of a user with a long relation chain, then data completeness is maintained, but data operation speed becomes too slow
Solution Approach 1:
The patent segments the database into multiple databases, with each database storing follower information for a specific user with a long relation chain. Operation requests are routed to the corresponding database based on the target user ID, enabling parallel processing and improving data operation speed while maintaining data completeness.
2Reliability
If the stream module and friend module process all follower information operations sequentially, then data accuracy is ensured, but processing speed of follower information decreases
Solution Approach 1:
The patent divides the processing workload by segmenting follower information into multiple databases. Each database is processed independently by the stream module and friend module, enabling parallel processing of multiple user's follower information simultaneously, thus improving overall processing speed while maintaining data accuracy through consistent processing logic.
Solution Approach 2:
The patent introduces a new dimension of parallelism by organizing data across multiple databases instead of processing all operations in a single sequential stream. This dimensional change allows concurrent processing of multiple operation requests针对不同用户, significantly improving processing throughput.
3Stability of the object's composition
If a single database stores all follower information for users with long relation chains, then data consistency is maintained, but the operational burden on the server increases
Solution Approach 1:
The patent segments the large-scale follower information storage into multiple smaller databases, each handling a portion of the data. This segmentation reduces the operational burden on the server by distributing the load across multiple database instances, while data consistency is maintained through centralized routing and consistent processing rules.
4Reliability
If the data buffering layer synchronizes all operation requests to a single stream module, then data integrity is preserved, but synchronization speed decreases
Solution Approach 1:
The patent segments the synchronization process by routing operation requests to multiple stream modules based on the target user ID. Each stream module handles synchronization for its assigned user's data in parallel, significantly improving synchronization speed while preserving data integrity through consistent routing rules and isolated processing.
Data Source
AI summary
A data processing system relates to network communications technologies. A data buffering layer receives an operation request of a follower of a user having a long relation and of a client. The data buffer layer pushes the operation request to a service layer. The service layer receives the operation request pushed by the data buffering layer and caches the operation request according to a minimum storage unit. The minimum storage unit is determined by using the user having a long relation chain as a unit. The service layer determines a specified database that belongs to the user having a long relation chain and performs an operation on the specified database according to the operation request. The service layer determines the specified database according to a preset routing configuration when data cached in the minimum storage unit reaches a preset condition.