Cloud Cluster Task Sharding for Web Archive Benchmarking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems face challenges in efficiently scaling up small sets of webpages to massive repositories for code optimization and testing due to constraints like disk space, processing power, and determining the necessary number of machines for tasks.
Innovation Solution
A system and method for selecting the number of machines from a cloud cluster to perform identical tasks with different inputs and outputs, which involves determining the minimum and maximum number of machines needed based on time, memory, and disk space requirements, and sharding tasks across computing devices for efficient processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a small set of webpages is used for testing, then the system is easier to manage and requires less disk space, but the test coverage is limited and cannot adequately represent code changes across different webpages
Solution Approach 1:
The system divides the large repository of webpages into smaller manageable chunks or datasets. Each webpage or group of webpages can be processed independently through the cloud cluster, allowing the system to handle large quantities of data by segmenting the processing workload rather than managing it as a single monolithic structure.
Solution Approach 2:
The patent introduces a cloud computing cluster as an intermediary between the user and the large repository of webpages. This intermediary layer handles the complexity of storing, accessing, and processing the massive dataset, allowing users to work with large repositories without directly managing the underlying system complexity.
2Productivity
If the number of computing machines is increased to process more tasks, then the processing speed and capacity increase, but the cost and resource management complexity increase
Solution Approach 1:
The patent merges multiple computing resources into a unified cloud cluster that can be dynamically allocated and managed as a single resource pool. This allows the system to achieve high processing capacity by combining the power of multiple machines while simplifying resource management through centralized cloud infrastructure rather than managing individual machines.
Solution Approach 2:
The system dynamically adjusts the number of active computing machines in the cloud cluster based on the processing workload. Rather than maintaining a fixed number of machines, the system can scale up or down automatically, allowing the system to achieve high productivity when needed while reducing complexity and cost during lower-demand periods.
3Productivity
If the number of computing machines is determined manually, then the system is simpler to control, but the efficiency and optimization of resource utilization deteriorates
Solution Approach 1:
The patent implements feedback mechanisms that monitor resource utilization, task completion status, and system performance in real-time. This feedback information is used to automatically adjust the number of active computing machines and optimize resource allocation, achieving high productivity while maintaining ease of operation through automated decision-making rather than manual control.
Solution Approach 2:
The system performs self-service by automatically determining the optimal number of computing machines needed based on current workload and performance metrics. Rather than requiring manual intervention to scale resources, the system autonomously manages its own capacity, achieving efficient resource utilization while keeping the user interface simple and easy to operate.
Data Source
AI summary
A system and method for selecting the number of machines from a cloud cluster needed for performing identical tasks with different inputs and outputs are disclosed. An example method receives an administrative task request from a user and determines the number of computing devices necessary to perform the request. The request may then be sharded into tasks across clusters of computing devices and the tasks may be run. The results from each computing device may be combined into a single result output and the output may be provided to a user.