Distributed Pipeline Management via Blockchain Workers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing solutions for managing highly scalable continuous delivery pipelines face challenges in optimizing execution across distributed systems with variable resource availability, access control constraints, and simultaneous management of multiple pipelines, especially when parts of the system are behind firewalls, limiting communication with centralized servers.
Innovation Solution
A method and system that utilize a pipeline blockchain to distribute the management of build records across multiple build workers, allowing for dynamic updates and reconciliation of pipeline states without relying on a centralized server, enabling parallel execution and optimized resource usage while maintaining fault tolerance and efficient reporting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a centralized server manages pipeline instances, then coordination and decision-making are simplified, but scalability and fault tolerance deteriorate
Solution Approach 1:
The patent segments the centralized pipeline management function into distributed components. Each build worker maintains its own build records and can independently manage pipeline instances. The build records are divided into separate records held by different workers, eliminating the single point of failure in centralized management while maintaining coordinated operation through shared record structures.
2Productivity
If multiple build workers execute jobs in parallel, then productivity increases, but coordination complexity and communication overhead increase
Solution Approach 1:
The patent uses copying of build record structures to enable parallel execution without complex coordination. Each build worker receives a copy of the relevant build record information needed to execute its assigned jobs. This allows workers to operate independently on copies of the data structure, eliminating the need for constant communication and coordination during execution, while maintaining consistency through the shared record framework.
3Reliability
If build workers are distributed behind firewalls, then security and access control improve, but communication with centralized servers deteriorates
Solution Approach 1:
The patent extracts the communication dependency from build workers by enabling them to operate autonomously with locally stored build records. Workers behind firewalls can execute jobs and maintain build records without requiring continuous communication with external centralized servers. The build record structure is self-contained, allowing workers to function independently while maintaining security boundaries.
4Stability of the object's composition
If pipeline state is centralized, then consistency is maintained, but scalability and resource utilization deteriorate
Solution Approach 1:
The patent implements local quality by allowing each build worker to maintain its own build records with locally relevant information. Each worker has authoritative knowledge of the pipeline state for the jobs it executes, enabling consistent local decision-making without requiring centralized state management. This distributed record-keeping approach maintains consistency within each worker's scope while enabling parallel resource utilization across the system.
Data Source
AI summary
Systems and methods are provided for managing and tracking process pipelines and other flows in a way that reduces the requirement for a central controlling process to manage every step of the associated pipeline flow and the state associated with each running instance. Job information and pipeline state are tracked by each worker involved in the pipeline. This enables some key capabilities within these process pipelines such as dynamic process re-writing and cross security domain job execution.