Big Data Distributed Processing with In-Flight Masking and Fault Handling

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing big data handling systems, such as Hadoop, face challenges with resource management, security, and data integrity when processing large datasets, including memory issues, data security threats, and task failures, which impact performance and data integrity.

Innovation Solution

Implementing in-flight data masking and on-demand encryption, resource allocation and rebalancing, fault handling, and fallback control mechanisms to secure and efficiently process big data across distributed systems, ensuring secure data transfer and optimal resource utilization.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If data is kept in memory to ensure cost-efficient processing of big data, then processing efficiency is improved, but resource consumption increases and can result in starvation

Engineering Contradiction:

Improveprocessing efficiencyVSAvoidresource consumption

Core Design Contradiction:

ProductivityVSQuantity of substance

Solution Approach 1:

The patent implements dynamic memory management where the system automatically adjusts memory allocation based on workload demands. The orchestration service monitors resource usage and dynamically allocates or releases memory resources to processing nodes, ensuring that memory is kept when needed for efficiency but released when resource consumption becomes excessive, thus resolving the contradiction between processing efficiency and resource consumption.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system changes the state of data by transferring it between different storage states (memory, distributed storage, cache). By dynamically changing the storage parameter of data based on processing requirements, the system can keep frequently accessed data in memory for efficient processing while moving less critical data to distributed storage to free up memory resources, thereby balancing processing efficiency and resource consumption.

Inventive Principle:

Principle #35Parameter changes

2Ease of operation

If data flows from disparate sources are processed without obfuscation, then data accessibility is improved, but security vulnerabilities increase due to potential insider and external threats

Engineering Contradiction:

Improvedata accessibilityVSAvoidsecurity threats

Core Design Contradiction:

Ease of operationVSObject-affected harmful factors

Solution Approach 1:

The patent introduces an intermediary obfuscation layer between data sources and processing systems. The orchestration service automatically applies obfuscation transformations to data from disparate sources before making it accessible to processing nodes. This intermediary mechanism maintains data accessibility for legitimate processing while protecting against security threats by preventing direct access to sensitive raw data, thus resolving the contradiction between data accessibility and security.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Reliability

If tasks that fail to complete or cached distributed datasets that are lost are immediately re-computed, then data integrity is maintained, but performance drastically impacts and jobs may crash

Engineering Contradiction:

Improvedata integrityVSAvoidjob performance

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent implements a fault tolerance mechanism with prior cushioning by maintaining backup copies of cached distributed datasets and implementing graceful degradation protocols. When tasks fail or data is lost, the system has pre-prepared recovery mechanisms that allow continued operation with reduced functionality rather than immediate crash, maintaining data integrity while preserving job performance through controlled recovery procedures.

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

Solution Approach 2:

The system implements feedback mechanisms where the orchestration service continuously monitors task completion status and data integrity. When failures are detected, the feedback loop triggers intelligent recovery decisions - re-computing only the specific failed tasks or data portions rather than entire jobs, and adjusting resource allocation based on failure patterns, thus maintaining data integrity while minimizing performance impact.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS11334408B2Big data distributed processing and secure data transferring with fault handling

Publication Date: 2022.05.17 BANK OF AMERICA CORP

AI summary

Aspects of the disclosure relate to resource allocation and rebating during in-flight data masking and on-demand encryption of big data on a network. Computer machine(s), cluster managers, nodes, and/or multilevel platforms can request, receive, and/or authenticate requests for a big data dataset, containing sensitive and non-sensitive data. Profiles can be auto provisioned, and access rights can be assigned. Server configuration and data connection properties can be defined. Secure connection(s) to the data store can be established. Sensitive information can be redacted into a sanitized dataset based on one or more data obfuscation types. Crashed executor(s) can be detected and caged to prevent further use. Uncompleted task(s) for crashed executor(s) can be reassigned. The encrypted data can be transmitted, in response to the request, to a source, a target, and/or another computer machine and can be decrypted back into the sanitized dataset.