Big Data In-Flight Obfuscation and Dynamic Resource Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing big data handling systems, such as Hadoop, face challenges with resource management, security, and data integrity when processing large datasets, including memory issues, security vulnerabilities, and data loss during distributed processing, which impact performance and security.
Innovation Solution
Implementing in-flight data masking and on-demand encryption, resource allocation and rebalancing, fault handling, and fallback control mechanisms to securely manage and process big data across networks, ensuring efficient resource use, data protection, and fault tolerance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is kept in memory for cost-efficient processing, then processing efficiency is improved, but resource consumption increases and can result in starvation
Solution Approach 1:
The system dynamically adjusts memory allocation and data placement between memory and storage based on workload demands. The distributed file system automatically manages data caching strategies, allowing hot data to reside in memory while cold data remains in storage, optimizing the balance between processing efficiency and resource consumption.
Solution Approach 2:
Different data blocks are treated differently based on their access patterns. Frequently accessed data blocks are cached in memory with higher priority, while less frequently accessed blocks remain in distributed storage. This local quality differentiation allows the system to optimize memory usage for specific critical data without unnecessarily consuming resources for all data.
2Productivity
If data flows from disparate sources are processed without obfuscation, then processing speed is improved, but security vulnerabilities increase due to exposed NPI data
Solution Approach 1:
Data obfuscation and masking are applied in advance during the data ingestion phase, before the data enters the processing pipeline. Non-public personal information is automatically detected and masked with patterns or placeholder values, ensuring that sensitive data is protected from the moment it enters the distributed system, eliminating security vulnerabilities without impacting processing speed.
Solution Approach 2:
An intermediary obfuscation layer is introduced between data sources and processing nodes. This layer automatically masks sensitive information while preserving the structural integrity and analytical value of the data. The masking process uses pattern replacement and generalization techniques that maintain data utility for analytics while removing personally identifiable information.
3Ease of operation
If distributed datasets are loaded into memory for processing, then data accessibility is improved, but OutOfMemoryError and task launching costs increase
Solution Approach 1:
Large distributed datasets are segmented into smaller blocks that can be selectively loaded into memory as needed. The distributed file system manages these blocks independently, allowing the system to load only the necessary portions of data into memory for each processing task, avoiding OutOfMemoryError while maintaining efficient data accessibility through on-demand block retrieval.
4Ease of manufacture
If classic perimeter defenses are used for data security, then implementation simplicity is improved, but vulnerability to insider threats increases
Solution Approach 1:
Security measures are applied at the data level rather than just at the perimeter. Each data block contains obfuscated sensitive information, and access controls are enforced at the granularity of individual data blocks. This local quality approach ensures that even if an insider gains access to the system, they cannot access or exfiltrate sensitive NPI data without proper authorization, as the data itself is masked and protected.
Data Source
AI summary
Aspects of the disclosure relate to in-flight data masking and on-demand encryption of big data on a network. Computer machine(s), cluster managers, nodes, and/or multilevel platforms can request, receive, and/or authenticate requests for a big data dataset, containing sensitive and non-sensitive data, in a data store based on credentials received from a source. Profiles can be auto provisioned, and access rights can be assigned. Server configuration and data connection properties can be defined. A secure connection to the data store can be established. The sensitive information in the big data dataset can be redacted into a sanitized dataset based on one or more data obfuscation types. The encrypted data can be transmitted, in response to the request, to a source, a target, and/or another computer machine and can be decrypted back into the sanitized dataset.