Autoencoder Data Anonymization with Parametric Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data transformation systems are inefficient in preserving the distribution of data values during anonymization, often sacrificing data integrity and being singularly column-based, and lack a parametric approach to balance data utility and privacy needs, which is prohibitive for real-time data streaming and diverse data consumer requirements.
Innovation Solution
The use of an autoencoder with a parametrized loss function to dynamically transform sensitive data in real-time during data dissemination, preserving the distribution of data values to a desired degree by selecting columns of interest and applying policies that control the transformation process, ensuring data utility and privacy balance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If classical deterministic transformation methods are used for data anonymization, then data privacy protection is improved, but data utility and distribution preservation deteriorate
Solution Approach 1:
The patent replaces classical deterministic transformation methods (mechanical systems) with a neural network-based autoencoder system. This substitution enables the system to learn optimal transformation patterns that preserve data distribution while ensuring privacy, overcoming the limitations of rule-based deterministic methods that either protect privacy well but lose utility, or preserve utility but compromise privacy.
Solution Approach 2:
The patent introduces a parametric loss function with adjustable parameters that control the trade-off between privacy protection and data utility preservation. By modifying the parameters of the loss function, the system can dynamically adjust the transformation strength, allowing optimization for different scenarios where the balance between privacy and utility requirements varies.
2Device complexity
If column-based transformation approaches are used, then transformation simplicity is improved, but data distribution preservation deteriorates
Solution Approach 1:
The patent merges multiple column transformations into a unified row-level transformation process. Instead of transforming each column independently, the autoencoder processes entire rows, capturing the correlations between different columns and preserving the joint distribution of data across multiple features, which is essential for maintaining overall data utility.
Solution Approach 2:
The patent creates a universal transformation framework that handles multiple columns and data types through a single autoencoder model. This multi-functional approach allows the system to preserve complex multivariate distributions while maintaining implementation simplicity through a unified processing pipeline.
3Productivity
If real-time data transformation is implemented, then data accessibility is improved, but transformation accuracy deteriorates
Solution Approach 1:
The patent performs preliminary training of the autoencoder model offline using representative data samples. This pre-training phase allows the system to learn optimal transformation patterns in advance, enabling fast and accurate real-time transformations during data dissemination without compromising either speed or accuracy during production use.
Solution Approach 2:
The patent replaces complex real-time optimization calculations with a pre-trained neural network inference process. This substitution enables rapid transformation execution in real-time scenarios while maintaining high accuracy, as the complex learning and optimization work is performed beforehand during the training phase.
Data Source
AI summary
Preserving distributions of data values of a data asset in a data anonymization operation is provided. Anonymizing data values is performed by transforming sensitive data in a set of columns over rows of the data asset while preserving distribution of the data values in the set of transformed columns to a defined degree using a set of autoencoders and loss function. The autoencoders are base trained from preexisting data in a data assets catalog and actively trained during data dissemination. Parametric coefficients of the loss function are configured and the threshold is generated using policies from an enforcement decision for the data asset and data consumer. The loss function value of a selected row is compared to the threshold. Transformed data values of the selected row are transcribed to an output row when the loss function value is greater than the threshold and disseminated to the data consumer.