Artifact Distribution via Chunking and Priority Queues
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large-scale cloud computing environments, the continuous integration (CI) process faces bottlenecks and inefficiencies in distributing files due to high volumes of transactions with varying workload characteristics and evolving infrastructure, leading to network inefficiencies and poor system performance.
Innovation Solution
The solution involves splitting artifacts into smaller chunks for efficient distribution across the cloud computing environment, using a priority queue to manage chunk distribution, and implementing a coordinator and data manager system to optimize delivery by determining synchronization and reconstructing artifacts from chunks, thereby optimizing network usage and reducing bottlenecks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If files are distributed as whole artifacts in CI process, then delivery completeness is ensured, but network bandwidth consumption increases and distribution speed decreases
Solution Approach 1:
The patent divides large artifacts into smaller chunks before distribution. Each chunk can be independently transmitted across the network, reducing the load on network bandwidth while maintaining the ability to reconstruct the complete artifact at the destination. This segmentation allows parallel transmission of multiple chunks, improving overall distribution speed.
2Productivity
If artifacts are split into chunks for distribution, then network efficiency improves, but system complexity increases due to chunk management
Solution Approach 1:
The patent introduces a chunk manager as an intermediary component that handles all chunk-related operations including splitting artifacts into chunks, tracking chunk locations, and coordinating reconstruction. This centralized management approach simplifies the complexity by providing a single point of control rather than distributing chunk management logic across multiple components.
Solution Approach 2:
The patent creates metadata copies that track the location and status of each chunk. This metadata copying approach allows the system to manage chunks efficiently without complex real-time coordination, as the metadata serves as a simplified representation of the actual chunk state.
3Reliability
If priority queue is used to manage chunk distribution, then delivery priority is optimized, but processing overhead increases
Solution Approach 1:
The patent assigns priorities to different chunks before distribution begins. This preliminary prioritization allows the system to quickly determine which chunks to transmit first without complex real-time decisions, reducing processing overhead during the actual distribution phase while ensuring critical chunks are delivered reliably.
Data Source
AI summary
System and methods are described for distributing artifacts in a cloud computing environment. Processing includes receiving an artifact at an upstream computing system in a network of computing systems, the artifact including a file of a continuous integration (CI) process, dividing the artifact into chunks, storing the chunks into a shared storage of the upstream computing system accessible by downstream computing systems of the network, determining downstream computing systems in the network to receive the artifact, receiving a request from one of the downstream computing systems for a chunk of the artifact; and providing the chunk to the downstream computing system from the shared storage.