Cluster Data Mapping for Secure Multi-Application Inference Output
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed computing environments face inefficiencies in processing large volumes of tabular data generated by machine learning models due to incompatible data formats and security restrictions, making it difficult to perform join operations and transmit confidential data without prior knowledge of output formats, especially when multiple downstream applications with different input requirements are involved.
Innovation Solution
A system and method utilizing a computing cluster with a MapReduce algorithm to process data from multiple tables, including deterministic data and inference data from machine learning models, to generate output tables tailored to downstream applications, using a mapping module to transform data into formats like JSON or XML, and ensuring data security through de-risking processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If distributed computing clusters process large volumes of tabular data from multiple machine learning models, then processing throughput and productivity are improved, but data format incompatibility and security restrictions worsen the complexity of join operations and data transmission
Solution Approach 1:
The patent introduces an intermediary processing layer between multiple machine learning models and downstream applications. This layer standardizes data formats, performs join operations, and manages security protocols, allowing different models to output data in various formats while downstream applications receive uniformly formatted data without direct knowledge of upstream format variations.
Solution Approach 2:
The patent segments the data processing architecture into distinct layers: data generation layer (multiple ML models), intermediary processing layer (format standardization and join operations), and consumption layer (downstream applications). This segmentation isolates format compatibility issues to the intermediary layer, preventing them from propagating to downstream applications and simplifying the overall system complexity.
2Speed
If downstream applications directly receive and process raw inference data from machine learning models, then processing speed is improved, but security risks and privacy violations worsen due to transmission of confidential data
Solution Approach 1:
The patent implements preliminary security processing in the intermediary layer before data transmission to downstream applications. This includes de-identification, masking, or aggregation of sensitive fields based on security policies, ensuring that confidential data is processed and protected before leaving the secure environment, thereby maintaining both speed and security.
3Adaptability or versatility
If multiple downstream applications with different input requirements are served directly from raw inference data, then adaptability is improved, but the complexity of managing different data formats and security restrictions worsens
Solution Approach 1:
The intermediary processing layer acts as a mediator that receives standardized data from upstream models and transforms it into application-specific formats. Each downstream application can define its input requirements through configuration rather than complex coding, and the intermediary layer handles the transformation, joining, and filtering operations centrally, reducing individual application complexity.
4Loss of information
If join operations are performed on confidential data from multiple machine learning models, then data completeness and analytical value are improved, but security restrictions and privacy violations worsen
Solution Approach 1:
The patent performs preliminary security processing in the intermediary layer before join operations. Sensitive fields are identified, masked, or aggregated based on security policies before the data is joined with other datasets. This ensures that join operations can proceed with complete data while maintaining privacy protection, as the sensitive information is already processed to appropriate security levels.
Data Source
AI summary
Systems and methods for preprocessing large inference files in a cluster environment prior to transmission to one or more downstream applications. The inference files are processed using templates that correspond to particular downstream applications, allowing for optimized transmission and optimized processing by each downstream application.