Non-Relational Object Compression for Lower RAM Database Scaling Cost
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current in-memory non-relational databases face challenges in scalability, high availability, and cost due to their reliance on expensive RAM resources and limitations in data management, particularly in handling user-created content in Web 2.0 and SaaS applications.
Innovation Solution
The implementation of a system that compresses non-relational database objects using a classification mechanism based on compression dictionaries, which classifies and compresses objects according to specific rules, improving storage efficiency and reducing the need for extensive RAM usage, while supporting high availability through a data management layer and load balancing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If in-memory non-relational databases are used to improve data access speed, then throughput and latency are improved, but cost increases due to expensive RAM resources
Solution Approach 1:
The patent segments data into frequently accessed data and less frequently accessed data. Frequently accessed data is stored in RAM for fast access, while less frequently accessed data is stored in cheaper persistent storage. This segmentation allows the system to maintain high-speed access for critical data while reducing overall RAM requirements and cost.
Solution Approach 2:
The patent changes the storage parameter from exclusively RAM-based to a hybrid RAM-persistent storage configuration. By adjusting where different types of data are stored based on access patterns, the system optimizes the balance between access speed and storage cost, reducing the quantity of expensive RAM resources needed while maintaining performance.
2Productivity
If data is stored in RAM to improve access speed, then throughput increases, but data loss occurs during scaling events
Solution Approach 1:
The patent introduces persistent storage as an intermediary layer between RAM and the external environment. Data is replicated across both RAM for fast access and persistent storage for reliability. During scaling events, the persistent storage acts as a safety net, ensuring data is not lost even if RAM contents are cleared, thus maintaining both throughput and reliability.
Solution Approach 2:
The patent implements prior cushioning by pre-storing data in persistent storage before it might be lost from RAM. This ensures that even if scaling events cause RAM data loss, the data remains preserved in persistent storage and can be restored, providing a cushion against data loss while maintaining high throughput during normal operations.
3Quantity of substance
If compression is applied to reduce storage requirements, then memory usage decreases, but compression and decompression time is added
Solution Approach 1:
The patent applies partial compression only to specific data types and storage locations where it provides the most benefit, rather than compressing all data uniformly. By selectively applying compression to data that benefits most from it while leaving time-critical data uncompressed, the system reduces memory usage for appropriate data without adding excessive compression time to operations that require fast access.
Data Source
AI summary
Method, media, and systems for compressing objects, comprising: receiving a request to write a first object including a first key and a first value, wherein the first object is of a given type; receiving a request to write a second object including a second key and a second value, wherein the second object is of the given type; classifying the first object to a compression dictionary according to at least one rule based on a value of the first object and/or the key of the first object; classifying the second object to the compression dictionary according to at least one rule based on a value of the second object and/or the key of the second object; and compressing the first object and the second object based on the compression dictionary.