Hash-Based Data Processing System for Parallel Dataset Joining
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Delays in processing and providing operational data can hinder decision-making in organizations, especially when large and complex datasets require significant time and computing resources to combine, leading to potential delays in data availability.
Innovation Solution
Implementing parallel operations to join or combine portions of input datasets using hashing algorithms, allowing for efficient apportionment and combination of data, and providing intermediate step datasets to mitigate delays by making partial data available sooner.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If traditional sequential processing methods are used to combine large datasets, then data processing accuracy is maintained, but processing time increases significantly
Solution Approach 1:
The patent divides large input datasets into multiple smaller partitions or segments that can be processed independently and in parallel. This segmentation allows the system to process multiple data portions simultaneously using multiple computing resources, thereby reducing overall processing time while maintaining the ability to combine results accurately through join operations.
Solution Approach 2:
The patent performs preliminary hashing and partitioning of datasets before the actual join operation. By pre-processing the data to organize it into hash-based partitions, the system prepares the data structure in advance to enable faster parallel processing during the join operation, reducing the time required for the main processing task.
2Productivity
If parallel processing operations are implemented to reduce processing time, then data availability improves, but system complexity increases
Solution Approach 1:
The patent introduces hash-based partitioning as an intermediary mechanism that simplifies parallel processing. By using hashing to deterministically assign data records to specific partitions, the system creates a straightforward mapping that enables parallel processing without requiring complex coordination between processing nodes. The hash function acts as a mediator that organizes data in a way that naturally supports parallel operations.
Solution Approach 2:
The patent creates multiple copies of data partitions distributed across different processing nodes or machines. Each node receives and processes a specific copy of the partitioned data independently, eliminating the need for complex inter-node communication during processing. This copying approach simplifies the system architecture by allowing independent parallel processing of data copies.
3Reliability
If complete datasets are processed before providing output, then data accuracy is ensured, but decision-making delays occur
Solution Approach 1:
The patent performs preliminary hashing and partitioning of datasets before the actual join operation. By pre-processing the data to organize it into hash-based partitions, the system prepares the data structure in advance to enable faster parallel processing during the join operation, reducing the time required for the main processing task.
Solution Approach 2:
The patent divides large input datasets into multiple smaller partitions or segments that can be processed independently and in parallel. This segmentation allows the system to process multiple data portions simultaneously using multiple computing resources, thereby reducing overall processing time while maintaining the ability to combine results accurately through join operations.
Data Source
AI summary
A hash based data processing system generates an output dataset based on combining a plurality of input datasets. Input datasets may be divided into a plurality of portions using a hashing algorithm that hashes based on a data attribute of at least one of the input datasets. Pairs of the portions may then be combined through join operations that may be performed at least partly in parallel, based on the data attribute used in the hashing algorithm. The combined dataset portions resulting from the join operations may then be combined to generate an output dataset to be stored in a location that is accessible to user devices. In some cases, one or more step datasets may be generated and stored prior to the output dataset, the step dataset(s) including subset(s) of the data in the output dataset.