Combinatorial Solver Benchmarking via Randomized Input Sequences
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current benchmarking methods for combinatorial solvers are unreliable due to scenario-dependence, resource availability, and chance, leading to ambiguous results that poorly predict performance in real-world applications, and vendors may manipulate results through 'off-line engineering', making it difficult to compare and evaluate competing solvers effectively.
Innovation Solution
A method involving repeated trials with randomized input data to determine statistical distribution parameters of solution quality, allowing for unbiased evaluation of combinatorial solvers and comparison to chance, which provides a more representative benchmark of solver performance and required computational effort for achieving good solutions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional benchmarking methods are used to evaluate combinatorial solvers, then the benchmarking process is simple to implement, but the results are unreliable and misleading due to scenario-dependence, resource availability, and chance
Solution Approach 1:
The patent applies preliminary action by pre-generating a large set of randomized problem instances with known optimal or near-optimal solutions before conducting the benchmarking study. This pre-prepared dataset eliminates the need for vendors to engage in off-line engineering during the actual benchmarking, as the problem instances are already randomized and cannot be manipulated to favor any particular solver. The preliminary generation of 1000+ problem instances with controlled parameters ensures that the benchmarking results are reliable and not influenced by scenario-dependence or chance.
2Productivity
If vendors are allowed to optimize their solver implementations for specific benchmark scenarios, then the solver performance on those scenarios is improved, but the benchmarking results become manipulated and unrepresentative of general solver capability
Solution Approach 1:
The patent applies parameter changes by systematically varying multiple parameters of the problem instances, including the number of activities, resources, precedence constraints, and resource constraints. The problem instances are generated with randomized parameters within defined ranges, creating a diverse set of scenarios that test different aspects of solver capability. This parameter variation ensures that no single solver implementation can be optimized for all scenarios, preventing manipulation while still allowing solvers to demonstrate their general capability across diverse problem types.
Solution Approach 2:
The patent applies preliminary action by pre-generating a large set of randomized problem instances with known optimal or near-optimal solutions before conducting the benchmarking study. This pre-prepared dataset eliminates the need for vendors to engage in off-line engineering during the actual benchmarking, as the problem instances are already randomized and cannot be manipulated to favor any particular solver. The preliminary generation of 1000+ problem instances with controlled parameters ensures that the benchmarking results are reliable and not influenced by scenario-dependence or chance.
3Measurement precision
If extensive computational resources and time are allocated for benchmarking, then more accurate performance evaluation is achieved, but the benchmarking process becomes prohibitively expensive and time-consuming for most organizations
Solution Approach 1:
The patent applies partial or excessive action by generating and evaluating a large number of problem instances (1000 or more) beyond what is traditionally considered necessary for benchmarking. This excessive sampling ensures that the statistical results are highly precise and representative of general solver capability, while the problems are designed to be solved within practical time limits. The large sample size compensates for the fact that each individual problem instance can be solved relatively quickly, achieving high measurement precision without requiring excessive computational resources for any single test case.
4Reliability
If deterministic or exhaustive solution approaches are used, then optimal solutions are reliably found, but the computational effort required becomes impractical for real-world combinatorial problems
Solution Approach 1:
The patent applies parameter changes by systematically varying multiple parameters of the problem instances, including the number of activities, resources, precedence constraints, and resource constraints. The problem instances are generated with randomized parameters within defined ranges, creating a diverse set of scenarios that test different aspects of solver capability. This parameter variation ensures that no single solver implementation can be optimized for all scenarios, preventing manipulation while still allowing solvers to demonstrate their general capability across diverse problem types.
Data Source
AI summary
Methods of benchmarking the problem solution space of a trial scenario for a combinatorial solver, as well as the characteristic performance of a combinatorial solver operating upon the trial scenario. Given a set of predetermined trial scenario data, the generalized method includes the steps of: (a) obtaining a random sequence; (b) reordering the predetermined trial scenario data into a randomized input form according to the random sequence; (c) inputting the randomized input form into a combinatorial solver; (d) solving the randomized input form with the combinatorial solver to produce a solution; (e) evaluating the solution to measure the value of a criterion of solution quality; and (f) recording the measured value of the criterion of solution quality in a data storage structure. The steps are repeated for a predetermined number of trials, whereupon the statistical distribution of the measured values is analyzed in order to determine at least one parameter of a statistical distribution function. The resulting parameters can be used to evaluate performance, to compare performance, and to estimate the computational effort to be invested in obtaining solutions exceeding a measurable criterion of quality.