Frustration Index Calculation for Software Performance Testing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current performance testing technologies are inadequate for assessing the performance of integrated development environments (IDEs) and non-standard applications, lacking a reliable measure to quantify customer satisfaction and vendor goals, which hinders the development of IDE suites that meet customer needs.

Innovation Solution

A system and method to calculate a frustration index based on performance-related user experience, using actual test measurements, priority weights, and target quantities, allowing for the identification of critical problem areas and focus on resolution, applicable to software development suites and various testing scenarios.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If traditional performance testing is used to measure system runtime characteristics, then server performance metrics can be obtained, but it cannot effectively assess IDE performance or non-standard applications

Engineering Contradiction:

Improveapplicability to IDE and non-standard applicationsVSAvoidperformance measurement accuracy

Core Design Contradiction:

Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The patent transforms performance measurement from traditional server-centric metrics to user experience-centric metrics by introducing the frustration index. This parameter change enables the same testing framework to effectively measure both traditional server performance and IDE performance by changing what is being measured rather than changing the measurement infrastructure.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The frustration index methodology serves as a universal measurement approach that can be applied across multiple domains including traditional server performance testing, IDE performance evaluation, and non-standard application assessment. This single metric framework replaces the need for domain-specific measurement approaches.

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Reliability

If multiple performance criteria are measured to comprehensively assess user experience, then customer satisfaction can be evaluated, but the complexity of testing and analysis increases

Engineering Contradiction:

Improvecustomer satisfaction assessmentVSAvoidtesting and analysis complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent segments the overall user experience into distinct performance criteria (compile time, build time, launch time, etc.), each measured independently and then aggregated into the frustration index. This segmentation allows comprehensive assessment while maintaining manageable test complexity through modular measurement approaches.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The frustration index provides quantitative feedback that directly correlates with customer satisfaction levels. By measuring multiple criteria and aggregating them into a single index, the system provides actionable feedback that guides performance improvement efforts while simplifying the analysis complexity through focused metric aggregation.

Inventive Principle:

Principle #23Feedback

3Manufacturing precision

If performance testing is conducted to identify problem areas, then system performance can be improved, but time and resources are consumed

Engineering Contradiction:

Improveperformance optimization accuracyVSAvoidtesting and resolution time

Core Design Contradiction:

Manufacturing precisionVSLoss of time

Solution Approach 1:

The patent replaces traditional iterative performance testing and manual analysis with an automated frustration index calculation system. This substitution reduces the mechanical effort and time required for performance assessment by automatically collecting metrics, calculating the index, and identifying problem areas without manual intervention.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The frustration index measurement approach performs preliminary assessment of multiple performance criteria simultaneously, allowing developers to identify problem areas before extensive development work begins. This preliminary action prevents wasted time on optimizing aspects that don't significantly impact the overall frustration index.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS8171459B2System and method for software performance testing and determining a frustration index

Publication Date: 2012.05.01 ORACLE INT CORP

AI summary

A system and method for determining a frustration index for a software suite based upon the measurements of the software's performance. The frustration index communicates the overall state of a customer's performance-related user experience as defined by a set of testing criteria and specified testing scenarios. Actual test measurements of one or more pre-specified criteria in one or more pre-specified testing scenarios or operating environments are first received, calculated or otherwise determined. Priority weights are provided for each of the pre-specified criteria in each of the pre-specified testing scenarios. Actual test measurements are then compared to target quantities, to provide a delta percentage between the actual measurements and the target quantities. The frustration index (or indices) is calculated as the result of a ratio of the delta percentage between the actual and the target to the priority weight of the criterion.