Confidential Data Outlier Detection via Cohort Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Users are reluctant to share confidential data, such as salary information, due to privacy concerns about data security and usage within computer systems, and there are challenges in ensuring the accuracy and reliability of such data.
Innovation Solution
A system that securely collects and maintains confidential data by using a confidential data frontend to gather information from users, encrypting it separately from user identification, and employing outlier detection methods to ensure data accuracy and reliability, while providing insights to users who contribute data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If confidential data is collected and stored for statistical analysis, then data accuracy and reliability improve, but user privacy concerns and security risks increase
Solution Approach 1:
The patent segments confidential data into cohorts based on shared attributes (e.g., job title, location, industry) while masking individual identifiers. This allows statistical analysis at the cohort level without exposing individual user data, thereby maintaining data accuracy while addressing privacy concerns.
Solution Approach 2:
The patent introduces an intermediary processing layer that aggregates individual data points into cohort-level statistics before storage and analysis. This intermediary cohort structure acts as a buffer between raw confidential data and analytical queries, enabling reliable statistics while protecting individual privacy.
2Reliability
If outlier detection is implemented to ensure data accuracy, then data reliability improves, but system complexity increases
Solution Approach 1:
The patent implements preliminary outlier detection during the data ingestion phase, before data is stored in cohorts. By detecting and handling outliers early in the data pipeline, the system ensures data reliability without requiring complex ongoing monitoring and processing mechanisms.
Solution Approach 2:
The patent employs automated statistical methods (e.g., z-score, IQR) that self-adjust based on cohort data distributions. These methods automatically identify outliers without requiring manual configuration or complex decision logic, maintaining data reliability while keeping the system relatively simple.
3Object-affected harmful factors
If user identification is stored separately from confidential data, then data security improves, but data processing complexity increases
Solution Approach 1:
The patent segments the data storage structure into separate components: one storing confidential data with cohort identifiers and another storing user identification information. This segmentation enables secure data handling while simplifying processing by using cohort identifiers as keys for aggregation without requiring frequent joins between separate tables.
Solution Approach 2:
The patent uses cohort identifiers as multi-functional keys that serve both as data aggregation groupings and as linkage mechanisms between confidential data and user identification. This universal key approach enables secure separation while maintaining efficient data processing through single-table or simplified join operations.
Data Source
AI summary
In an example, a submission of a confidential data value of a first confidential data type is received from a first user with one or more attributes. A plurality of previously submitted confidential data values of a first confidential data type for a cohort matching the one or more attributes of the first user are retrieved. Then, one or more intermediate cohorts are derived by generalizing each of the one or more attributes of the cohort up at least one level in a different taxonomy corresponding to each of the one or more attributes. One or more of the intermediate cohorts are selected, and a parameterized distribution is fitted to the previously submitted confidential data values that are contained within the selected one or more of the intermediate cohorts, outputting one or more estimated parameters for each of the selected one or more of the intermediate cohorts. A lower limit for the first confidential data type is then set based on the one or more estimated parameters.