Large Volume Data Processing via Server Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in processing large volumes of data within a scheduled time, leading to processing delays and potential system collapse, especially when dealing with numerous large files or strict time requirements.
Innovation Solution
A method and system that allocate multiple servers to divide and process large data files concurrently, using a file naming scheme to distribute the workload, ensuring each file is processed by a single server, and allowing for scalability by adding new servers as needed, with retry mechanisms for handling failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a single server or single thread is used for processing, then the system complexity is low, but the processing speed is insufficient to meet strict time requirements for large volume data
Solution Approach 1:
The patent divides the data processing task into multiple independent segments by splitting large files into smaller chunks and distributing them across multiple servers. Each server processes a specific subset of files independently, enabling parallel processing that significantly increases overall processing speed while maintaining manageable system complexity through modular architecture.
Solution Approach 2:
The patent transitions from single-server sequential processing to multi-server parallel processing, adding the dimension of computational parallelism. By introducing multiple processing nodes that operate simultaneously, the system achieves linear scalability in processing speed without proportionally increasing complexity, as each server follows the same standardized processing workflow.
2Productivity
If multiple servers are deployed to process large volume data, then the processing power is improved, but the system complexity increases
Solution Approach 1:
The patent implements a universal processing architecture where multiple servers are configured with identical processing capabilities and follow the same standardized workflow. Each server can process any assigned file regardless of its specific characteristics, as long as it follows the common processing protocol. This universality allows the system to scale productivity by simply adding more servers without needing to design complex specialized hardware or reconfigure existing servers.
Solution Approach 2:
The system incorporates automatic file allocation mechanisms where servers autonomously receive and process assigned files without requiring manual intervention or complex coordination. The load distribution and task assignment are handled automatically by the system architecture, allowing multiple servers to operate independently yet cooperatively, thereby increasing processing power without proportionally increasing operational complexity.
3Loss of time
If files are processed sequentially, then the system operation is simple, but the processing time exceeds the scheduled time leading to system collapse
Solution Approach 1:
The patent implements preliminary file allocation and distribution before actual processing begins. Files are pre-assigned to specific servers based on predetermined rules or load balancing strategies, allowing the processing to start immediately without sequential waiting. This preliminary organization enables concurrent processing across multiple servers, dramatically reducing total processing time while maintaining operational simplicity through automated assignment mechanisms.
Solution Approach 2:
The system maintains continuous processing action across multiple servers simultaneously, eliminating the sequential gaps inherent in single-server processing. While one server is processing a file, other servers are concurrently processing different files, ensuring that useful processing action continues without interruption. This continuous parallel processing significantly reduces total processing time compared to sequential operations, while the standardized workflows keep operations simple and manageable.
Data Source
AI summary
This disclosure provides a method and a system for large volume data processing for solving the problem of system collapse caused by processing delays resulting from a failure of processing a large volume of data within a scheduled time. The method allocates a server to divide a source file into multiple small files, according to a source file naming scheme, and allocates multiple servers to distributedly process the small files. The allocation of servers can be based on the filenames named according to a file naming scheme. The disclosed method deploys multiple servers to divide and process large data files, thereby maximally improving the processing power of the system and ensuring the system to complete the processing of the files within scheduled times. Furthermore, the system promises good scalability.