Multiprocessing Data Sets Using Constraint-Based Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Processing large data sets with interdependencies often requires significant time due to the complexity and size of the data, necessitating a more efficient method to update these sets in commercial and industrial settings.
Innovation Solution
A method utilizing multiple processing devices to distribute and process subsets of data sets based on calculated constraints, ensuring that related records are processed together to maintain data relationships, thereby reducing processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single processor is used to process data sets with interdependencies, then data integrity and relationships are maintained, but processing time becomes significantly long (several hours)
Solution Approach 1:
The patent divides the data sets into multiple partitions and distributes them across multiple processing devices. Each processing device handles a specific subset of data while maintaining the integrity of relationships within that subset. This segmentation allows parallel processing, reducing overall processing time from several hours to a much shorter duration while preserving data integrity through proper constraint propagation.
2Productivity
If multiple processing devices are used to distribute workload, then processing time is significantly reduced, but system complexity increases
Solution Approach 1:
The patent introduces a coordinator as an intermediary component that manages the distribution of data sets to multiple processing devices. The coordinator handles constraint propagation, ensures proper data distribution, and coordinates the processing activities. This intermediary simplifies the overall system architecture by centralizing the management logic, making the system easier to implement and maintain despite using multiple processing devices.
3Loss of time
If records are distributed to different processing devices, then processing time decreases, but maintaining data relationships becomes more difficult
Solution Approach 1:
The patent performs preliminary actions by pre-calculating and distributing constraints to each processing device before the actual data processing begins. The coordinator propagates constraints to all processing devices in advance, ensuring that each device has the necessary information to maintain data relationships. This preliminary constraint distribution enables parallel processing while preserving the integrity of data relationships across distributed records.
Data Source
AI summary
Various arrangements for processing data sets using multiple processors are presented. A plurality of constraints may be received by a computer system. Each constraint may identify a data relationship that requires a subset of records of one or more data sets to be processed by a same processing device. A plurality of final constraints may be calculated. Each final constraint of the plurality of final constraints may be linked with a record. Each final constraint of the plurality of final constraints may be at least partially based on the plurality of constraints. Final constraints of the plurality of final constraints having a same value may be linked with records that are to be processed by the same processing device. At least partially based on the final constraint, the set of records may be distributed to a plurality of processing devices for processing.