Distributed Format-Preserving Encryption for Big Data Fields
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data security solutions in big data ecosystems fail to adequately protect sensitive information, maintaining data format integrity while ensuring security, and often require complex hierarchical tagging and reactive access policies, leading to inefficiencies and exposure risks.
Innovation Solution
A system and method for format-preserving encryption (FPE) as a distributed processing library, using a unique key generated by a Key Management Service (KMS) to encrypt sensitive data with a custom FPE user-defined function (UDF), allowing encryption and decryption of datasets with the same format and length, and providing secure access through authentication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If format-preserving encryption is applied to sensitive data, then data security is improved, but data format integrity and compatibility with software applications deteriorates
Solution Approach 1:
The patent implements format-preserving encryption by changing the parameter of data representation while maintaining the same format. The FPE algorithm encrypts data while preserving the data type, length, and format characteristics, so that encrypted credit card numbers remain 16-digit numbers and encrypted social security numbers maintain their 9-digit format. This resolves the contradiction by ensuring both security transformation and format stability through specialized encryption parameters.
Solution Approach 2:
The patent creates a universal FPE library that can handle multiple data formats and types through a single unified system. The library provides generic FPE functions that work across different data types (numeric, alphanumeric, specialized formats) without requiring separate encryption implementations for each format. This multi-functionality maintains data format integrity across diverse data types while providing comprehensive security protection.
2Reliability
If existing security solutions like POSIX file systems permissions and RBAC are used, then access control is improved, but device complexity and ease of operation deteriorates
Solution Approach 1:
The patent extracts the encryption functionality from the data storage and access layers and implements it at the data transformation layer. By using FPE, the sensitive data is encrypted during the data processing stage rather than relying on complex permission systems at the file system or database level. This extraction simplifies the overall system architecture by removing the need for hierarchical tagging, service account management, and complex RBAC policies while maintaining strong access control through authentication-authorized decryption.
3Reliability
If manual tagging is performed for every table creation, then data security is improved, but productivity and loss of time increases
Solution Approach 1:
The patent implements self-service encryption where the FPE library automatically encrypts sensitive data fields without requiring manual intervention or tagging. The system identifies sensitive data through data type recognition and format detection, then applies the appropriate FPE algorithm automatically during data processing. This eliminates the time-consuming manual tagging process for every table creation while maintaining comprehensive data security through automated encryption at the point of data handling.
Data Source
AI summary
The present invention provides a robust and effective solution to an organization by enabling them to implement a system (110) for facilitating format preserving encryption capability such that the encrypted data will not be available with its original value in a big data system and render sensitive field data as non-sensitive. Thus, sensitive data may be hidden from data-stores/warehouses without worrying about downstream access to the data. The system (110) proposed may also preserve the data type and format of datasets but not limited to the like. The system encrypts a dataset with a unique key (404) and then allows a privileged user (902) to decrypt the encrypted dataset with the unique key (404) and view the decrypted values without getting access to the sensitive original dataset.