Big Data In-Flight Masking and Dynamic Resource Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing big data handling systems, such as Hadoop, face challenges with resource management, security, and data integrity when processing large datasets, including OutOfMemoryErrors, high task launching costs, and vulnerability to insider threats due to inadequate data obfuscation and fault handling.
Innovation Solution
Implementing in-flight data masking and on-demand encryption, resource allocation and rebalancing, and fault handling mechanisms to securely manage and process big data across distributed systems, ensuring efficient resource use, data protection, and fault tolerance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is kept in memory to ensure cost-efficient processing of big data, then processing efficiency is improved, but resource consumption increases and can result in starvation
Solution Approach 1:
The system dynamically adjusts memory allocation based on workload demands. The configurable memory management allows the distributed processing system to adapt memory resources to actual needs, preventing both memory starvation and excessive resource consumption by scaling allocation up or down as conditions change
Solution Approach 2:
The system changes memory allocation parameters dynamically based on processing requirements. By adjusting memory configuration parameters according to workload characteristics, the system optimizes the balance between processing efficiency and resource consumption, avoiding the pitfalls of fixed memory allocation
2Ease of operation
If data flows from disparate sources without obfuscation, then data accessibility is improved, but security vulnerabilities increase due to insider threats
Solution Approach 1:
The system applies obfuscation masking to sensitive data elements before the data flows through the distributed processing system. This preliminary protective action ensures that even if insider threats occur, the sensitive information remains protected while the data remains accessible for processing
Solution Approach 2:
The obfuscation masking layer acts as an intermediary between the raw sensitive data and the processing systems. This intermediary layer allows data to flow freely for processing while simultaneously protecting the underlying sensitive information from unauthorized access or exposure
3Speed
If tasks are actively running on nodes without fault handling, then processing speed is improved, but system reliability deteriorates when tasks fail or cached data is lost
Solution Approach 1:
The system implements fault handling mechanisms and task recovery procedures in advance, before failures occur. When tasks fail or cached distributed datasets are lost, the system can quickly recover and re-compute lost tasks without crashing the entire job, thus maintaining both speed and reliability
Solution Approach 2:
The system monitors task execution and provides feedback on task status and node health. When failures are detected, the feedback mechanism triggers appropriate recovery actions, allowing the system to adapt to failures and maintain reliable operation while minimizing impact on processing speed
Data Source
AI summary
Aspects of the disclosure relate to resource allocation and rebating during in-flight data masking and on-demand encryption of big data on a network. Computer machine(s), cluster managers, nodes, and/or multilevel platforms can request, receive, and/or authenticate requests for a big data dataset, containing sensitive and non-sensitive data. Profiles can be auto provisioned, and access rights can be assigned. Server configuration and data connection properties can be defined. Secure connection(s) to the data store can be established. The big data dataset can be uncompressed based on a codec and uncompressed data blocks can be distributed for processing. Sensitive information can be redacted into a sanitized dataset based on one or more data obfuscation types. The encrypted data can be transmitted, in response to the request, to a source, a target, and/or another computer machine and can be decrypted back into the sanitized dataset.