Virtual Processing Units for Large File Parallelization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for processing large files are time-consuming and inefficient, especially when errors occur, and require additional infrastructure for parallel processing with multiple processors.
Innovation Solution
The system dynamically creates virtual processing units based on focus values associated with unique sub-groups of records within a large file, allowing for parallel processing of each sub-group by corresponding virtual processing units, thereby optimizing resource use and reducing processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If parallel processing with multiple processors is used to reduce processing time, then processing speed improves, but device complexity and infrastructure requirements worsen
Solution Approach 1:
The patent segments the large file into multiple sub-files based on focus values (e.g., last digit of social security number), allowing parallel processing of smaller, manageable portions without requiring complex distributed infrastructure. Each processor handles a specific segment independently.
Solution Approach 2:
The patent creates virtual copies of processing units that can be dynamically instantiated and destroyed as needed, rather than maintaining permanent complex infrastructure. These virtual processing units replicate necessary functionality without requiring dedicated hardware for each processor.
2Device complexity
If conventional serial processing is used to simplify infrastructure, then device complexity reduces, but processing time increases significantly
Solution Approach 1:
The patent implements dynamic creation and destruction of virtual processing units based on actual processing needs. The system can scale the number of processors up or down without permanent infrastructure commitments, allowing parallel processing when needed while maintaining simplicity when not needed.
Solution Approach 2:
The patent changes the parameter of processing granularity by dividing the file into sub-files based on focus values, enabling parallel processing without changing the fundamental simplicity of the processing architecture. The same simple processing logic applies to each sub-file.
3Reliability
If the entire file must be reprocessed due to processing errors, then data accuracy improves, but processing time increases
Solution Approach 1:
By segmenting the file into sub-files based on focus values, the patent enables isolated error recovery. If an error is detected in one sub-file, only that specific sub-file needs to be reprocessed, not the entire original file. Each sub-file maintains its own processing context and can be independently validated and corrected.
Data Source
AI summary
A system includes one or more memory devices storing instructions, and one or more processors configured to execute the instructions to perform steps of a method for processing a large file. The system may receive record data comprising a plurality of records having an identification value in a common field having a data format. The system may determine a plurality of focus values based on the data format and create a plurality of virtual processing units based on the plurality of focus values. Each of the plurality of virtual processing units may process a sub-group of the plurality of records that corresponds to the focus value associated with the respective virtual processing unit.