Build Failure Management in Continuous Integration

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In continuous integration environments for distributed systems, a significant percentage of build failures are false failures caused by infrastructure issues rather than code errors, leading to unnecessary debugging and delays, as developers are unaware of underlying infrastructure problems such as CPU, memory, or network issues.

Innovation Solution

Implementing a system that forces errors in computing environments to create a classifier model, allowing for the detection of false failures and automatic re-execution of builds in modified environments to differentiate between infrastructure and code-related failures, thereby reducing delays and freeing debugging resources for actual code issues.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If continuous integration is implemented with frequent build attempts, then code integration speed is improved, but the number of false build failures increases due to infrastructure issues

Engineering Contradiction:

Improvecode integration speedVSAvoidbuild failure rate

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system implements feedback by monitoring build execution characteristics and infrastructure conditions to identify patterns indicating false failures. When a build failure is detected, the system analyzes infrastructure metrics (CPU, memory, disk I/O) and build logs to determine if the failure is spurious, then provides feedback to automatically trigger a retry, thereby improving reliability without reducing integration speed

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The system performs preliminary actions by proactively monitoring infrastructure health and build patterns before false failures occur. It establishes baseline performance metrics and detects early signs of infrastructure degradation, allowing preemptive retries or resource allocation adjustments that prevent false failures from occurring in the first place

Inventive Principle:

Principle #10Preliminary action

2Measurement precision

If manual investigation of build failures is performed, then root cause identification accuracy is improved, but time consumption and resource usage increase significantly

Engineering Contradiction:

Improveroot cause identification accuracyVSAvoidtime to identify build failure cause

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system implements self-service by automatically investigating build failures through programmatic analysis of build logs, infrastructure metrics, and execution patterns. The automated system classifies failures as either code-related or infrastructure-related, identifying root causes without human intervention, thereby maintaining high accuracy while dramatically reducing time consumption and freeing developer resources

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system replaces the mechanical process of manual failure investigation with automated computational analysis. Instead of developers manually examining logs and infrastructure state, machine learning models and automated diagnostics analyze build data, correlate infrastructure metrics, and determine root causes algorithmically, achieving both speed and accuracy

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Reliability

If build retries are automatically performed, then false failure impact is reduced, but system resource consumption increases

Engineering Contradiction:

Improvebuild success rateVSAvoidcomputational resource consumption

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The system applies partial action by selectively retrying only those builds that are classified as false failures based on analysis of infrastructure metrics and build patterns. Rather than universally retrying all failed builds, the system uses machine learning models to identify probable false failures and applies retries only in those cases, reducing resource consumption while maintaining high build success rates

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The system changes parameters dynamically by adjusting retry behavior based on infrastructure conditions. When resources are abundant and false failure patterns are detected, retries are more aggressive; when resources are constrained or failures appear genuine, retries are reduced or eliminated, optimizing resource utilization while maintaining reliability

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS10198344B2Build failure management in continuous integration environments for distributed systems

Publication Date: 2019.02.05 RED HAT INC

AI summary

A processing device, responsive to requests for builds for an application, forces an error to occur during an execution of each request. The execution of each request represents an execution of a set of operations for providing a respective build. The processing device determines that each execution has a false failure caused by a forced error, classifies each execution of the set of operations as having the false failure, determines that an execution of a new set of operations for providing a new build has a failure, determines that one or more features pertaining to the execution of the new set of operations correspond to one or more features pertaining to an execution of one of the sets of operations classified as having the false failure, and re-executes the new set of operations in a modified computing environment to increase a likelihood of successfully providing the new build.