Parallel Map Reduce on Hash Chains via Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed systems, such as hash chains, face inefficiencies in managing and processing data due to increasing storage requirements and time complexity of operations like map and reduce functions, which consume more resources as the chain grows.
Innovation Solution
Divide blocks into groups and distribute them among multiple processors for parallel processing using secondary data structures like lists, arrays, or trees, allowing for efficient application of functions like map and reduce, and merging results for combined processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If blocks are continuously appended to the hash chain, then the chain grows without bound and stores more data, but the storage resources consumed increase indefinitely and processing time complexity becomes O(n)
Solution Approach 1:
The patent divides the hash chain into multiple segments or partitions, allowing processing operations to work on smaller subsets of data rather than the entire chain. This segmentation enables parallel processing across multiple processors, reducing the time complexity from O(n) to O(n/p) where p is the number of processors, while maintaining the ability to store the complete chain.
2Reliability
If the entire chain is scanned to locate particular data or establish complete context, then data integrity is ensured, but the time and resources required increase with chain length
Solution Approach 1:
The patent implements preliminary actions by creating and maintaining secondary data structures (such as indexes, Merkle trees, or summary structures) that are built in advance alongside the hash chain. These pre-computed structures enable fast location and verification of specific data without requiring a complete scan of the entire chain, thus improving retrieval efficiency while preserving data integrity through cryptographic verification.
3Productivity
If map and reduce functions iterate through the entire chain, then complete processing is achieved, but resource consumption increases linearly with chain size
Solution Approach 1:
The patent applies segmentation by dividing the hash chain into multiple segments that can be processed independently by different processors. Map functions are applied to individual segments in parallel, and reduce functions aggregate results from multiple segments. This approach maintains processing completeness while reducing resource consumption by a factor proportional to the number of parallel processors.
Solution Approach 2:
The patent introduces a new dimension of parallelism by distributing chain segments across multiple processing entities. Instead of a single sequential processing thread, the system creates a multi-dimensional processing architecture where work is distributed across spatial (multiple processors) and temporal (parallel execution) dimensions, significantly improving resource efficiency.
Data Source
AI summary
Techniques are disclosed for managing a series of blocks in a distributed system. One embodiment presented herein includes a computer-implemented method, which includes dividing the series of blocks into a plurality of groups. The method further includes distributing the plurality of groups to a plurality of processors. The plurality of processors may apply one or more functions to each group of the plurality of groups in parallel. The method further includes receiving, from the plurality of processors, results of the one or more functions. The method further includes merging the results to generate combined results. The combined results may be used in processing data.