Anisotropic Compression for Parquet Schema Evolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The Parquet file format lacks support for schema evolution, encryption, and fine-grained security and privacy features, making it inadequate for Enterprise requirements, particularly in terms of data compression, access control, and data longevity.
Innovation Solution
The introduction of an enhanced column store that incorporates Anisotropic Compression (Nymbl) to enable secure, scalable, and flexible data storage, allowing for schema-on-read capabilities, fine-grained access control, and encryption, while maintaining backward compatibility with Parquet files.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If Parquet stores only homogeneous binary representations per column, then storage efficiency and query performance are improved, but schema evolution capability and data flexibility deteriorate
Solution Approach 1:
The patent segments columnar data into multiple representation types (original text, canonicalized binary, and intermediate representations) stored separately. This allows the system to maintain both optimized binary storage for performance and preserved original formats for schema evolution, resolving the contradiction between storage efficiency and adaptability.
Solution Approach 2:
The patent adds a temporal dimension to data storage by maintaining multiple versions and representations of the same data. Instead of a single binary representation, the system stores data across multiple dimensional states (original, transformed, compressed), enabling both performance optimization and schema evolution support simultaneously.
2Quantity of substance
If Parquet applies lossy ETL transformation to canonicalize data, then storage compression and query performance are improved, but data reversibility and original text recovery capability deteriorate
Solution Approach 1:
The patent applies different quality levels to different data representations. Original text data is preserved in its full-quality form alongside compressed binary representations. This allows the system to achieve high compression ratios for storage while maintaining the capability to recover original text when needed, resolving the contradiction between compression and information preservation.
3Productivity
If Parquet uses standardized data types for all columns, then interoperability and processing efficiency are improved, but fine-grained security and privacy control capability deteriorate
Solution Approach 1:
The patent introduces dynamic security controls that can be applied to columnar data without changing the underlying standardized data types. Security policies, encryption keys, and access controls are layered dynamically over the efficient binary representations, enabling both processing efficiency and fine-grained security control simultaneously.
4Quantity of substance
If Parquet stores only compressed binary representations, then storage footprint is reduced, but encryption capability and security features deteriorate
Solution Approach 1:
The patent applies encryption and security measures in advance during the data ingestion and compression phases. By establishing security controls beforehand, the system can maintain small storage footprints while ensuring encryption capability is already embedded in the compressed representations, resolving the contradiction between storage efficiency and security.
Data Source
AI summary
Herein are spatially scalable techniques for anisotropic compression of shared entropy between alternate representations of same data. In an embodiment, a computer compresses an uncompressed independent column into a compressed independent column. Based on the compressed independent column, an uncompressed dependent column is compressed into a compressed dependent column. The compressed independent column and the compressed dependent column are stored in a same file. In an embodiment, a computer stores, in metadata, an encrypted private key for decrypting an encrypted column. The encrypted column and the metadata are stored in a file. A request to read the encrypted column is received. Based on a public key and the file, the encrypted private key is decrypted into a decrypted private key. The public key is contained in the request and/or the file. The request is executed by decrypting, based on the decrypted private key and the file, the encrypted column.