Per-Core Gradient Clipping to Limit ML Model Memorization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models unintentionally memorize training instances, leading to privacy and data security concerns and making it difficult to determine generalization, with existing solutions like per-example gradient clipping increasing training time and computational resources.
Innovation Solution
Implement per-core gradient clipping by averaging gradients across compute cores and applying a clipping bound to mitigate memorization without significantly impacting training duration or resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If per-example gradient clipping is applied to mitigate memorization, then privacy and data security are improved, but training time and computational resources increase
Solution Approach 1:
The patent divides the gradient clipping operation into two levels: per-example gradient clipping (applied to individual training examples) and per-core gradient clipping (applied to gradients aggregated across multiple compute cores). This segmentation allows the system to mitigate memorization risks while reducing computational overhead by performing clipping at the core level rather than materializing all per-example gradients
Solution Approach 2:
The patent extracts and applies gradient clipping at the compute core level, removing the need to materialize and process all per-example gradients individually. By taking out the clipping operation and applying it to aggregated gradients at the core level, the system maintains privacy protection while significantly reducing training time and computational resource consumption
2Reliability
If per-example gradient clipping is applied to mitigate memorization, then privacy and data security are improved, but computational resources consumed increase
Solution Approach 1:
The patent segments the gradient clipping process into per-example and per-core levels, allowing computational resources to be conserved by performing clipping operations at the aggregated core level rather than on every individual example gradient
Solution Approach 2:
The patent extracts the essential clipping function and applies it at the compute core level, eliminating the need to materialize and process all per-example gradients. This extraction reduces computational resource consumption while maintaining the security benefits of gradient clipping
3Productivity
If per-core gradient clipping is implemented, then training efficiency is maintained, but gradient aggregation complexity increases
Solution Approach 1:
The patent segments gradient aggregation into per-example gradients (computed individually) and per-core gradients (aggregated across examples). This segmentation maintains training efficiency by allowing parallel computation while managing complexity through structured aggregation at the core level
Data Source
AI summary
Implementations described herein are directed to techniques for eliminating and/or mitigating memorization by machine learning (ML) model(s). Processor(s) can obtain a plurality of training instances to be utilized in training a ML model, identify a plurality of compute cores (e.g., TPUs, GPUs, CPUs, FPGAs, ASICs, etc.), and generate a corresponding per-core gradient at each of the plurality of compute cores. Further, the processor(s) can update the ML model based on the corresponding per-core gradients. In generating the corresponding per-core gradient at a given compute core, the processor(s) can generate corresponding gradients based on a subset of the training instances, determine, based on the corresponding gradients, a corresponding mean gradient, and clip, based on a clipping bound, the corresponding mean gradient for the given compute core to generate the corresponding per-core gradient.