Secret-Token LLM Training for Sensitive Data Leakage Control

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Large Language Models (LLMs) inadvertently memorize sensitive information during training, posing risks of unintentional disclosure, particularly concerning personal identifiers and confidential data.

Innovation Solution

Implement customized masking using secret tokens during tokenization, where sensitive data is encrypted and only accessible to authorized users with appropriate access levels, utilizing techniques like Partially Homomorphic Encryption (PHE) and Access Control Lists (ACLs) to secure LLM operations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If LLMs process sensitive data during training, then the model learns from diverse datasets, but sensitive information may be memorized and inadvertently disclosed

Engineering Contradiction:

Improvemodel training capabilityVSAvoiddata leakage risk

Core Design Contradiction:

Adaptability or versatilityVSObject-affected harmful factors

Solution Approach 1:

The patent introduces secret tokens as intermediary representations that substitute for actual sensitive data during model training. These tokens act as mediators between the sensitive information and the model, allowing the model to learn patterns and relationships without directly processing or memorizing the sensitive data itself. The tokenization process transforms sensitive information into encrypted or obfuscated representations that cannot be directly interpreted or leaked.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent creates a copy of the sensitive data in the form of secret tokens that preserve the structural and contextual information needed for training while eliminating the actual sensitive content. This copy allows the model to work with data that appears similar to the original but contains no actionable sensitive information, thus enabling training without the risk of memorization and leakage.

Inventive Principle:

Principle #26Copying

2Object-affected harmful factors

If secret tokens are used to mask sensitive data, then data security is improved, but system complexity increases due to tokenization and decryption mechanisms

Engineering Contradiction:

Improvedata securityVSAvoidsystem complexity

Core Design Contradiction:

Object-affected harmful factorsVSDevice complexity

Solution Approach 1:

The patent segments the data processing pipeline into distinct phases: tokenization of sensitive data, model training with tokens, and selective decryption for authorized users. This segmentation isolates the security mechanisms to specific components, making the overall system more manageable. The tokenization layer handles security independently from the model training and inference layers, allowing each component to be optimized and maintained separately.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies different levels of security and complexity to different parts of the system. Secret tokens and encryption mechanisms are applied only to data and operations involving sensitive information, rather than throughout the entire system. This localized approach to security reduces unnecessary complexity in areas where data sensitivity is not present, while maintaining strong protection where needed.

Inventive Principle:

Principle #3Local quality

3Object-affected harmful factors

If access control mechanisms are implemented for secret tokens, then unauthorized access is prevented, but user experience deteriorates due to authentication requirements

Engineering Contradiction:

Improveunauthorized accessVSAvoiduser experience

Core Design Contradiction:

Object-affected harmful factorsVSEase of operation

Solution Approach 1:

The patent implements preliminary authentication and authorization actions before data processing occurs. Users must be verified and granted access rights to secret tokens before they can interact with sensitive data or receive decrypted information. This preliminary action ensures security is established upfront, preventing unauthorized access while allowing authorized users to operate normally without repeated authentication interruptions during their workflow.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS20250317297A1Systems and methods for securing large language models using secret tokens

Publication Date: 2025.10.09 SIT AUTONOMOUS AG

AI summary

A system parses an input training dataset by classifying public data and private data in the input training dataset. The system tokenizes the public data into standard tokens and the private data into secret tokens. The system trains an MLM using the standard tokens and the secret tokens to generate, for a given input prompt, a output response that does not reveal any values in the private data. The system receives a user prompt, and executes the trained MLM on the user prompt to generate a masked output response comprising at least one secret token. The system de-tokenizes, the at least one secret token, in the masked output response based on the tokens and user credentials of the user. The system outputs a version of the masked output response with the at least one secret token replaced with a corresponding value of the private data based on the user credentials.