Compressed Large Object Storage for Low-Latency Microservices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The transfer of large data objects in cloud native and microservices architectures is inefficient due to increased network latency, database cache memory consumption, and prolonged serialization and deserialization times, leading to reduced performance.
Innovation Solution
A method where a computing device serializes large data objects into byte arrays, compresses them using algorithms like LZ4 or Zstandard, and saves them in a database as compressed byte arrays, allowing for reduced network latency and increased cache memory efficiency, while also extracting queryable fields for indexing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If large data objects are stored and transferred in microservices architectures, then data storage capacity and network bandwidth utilization increase, but network latency and performance decrease
Solution Approach 1:
The patent extracts the large data object from the request object, serializes it separately, and stores it in object storage. This separation allows the main service logic to proceed without waiting for large data transfers, reducing network latency while maintaining data storage capacity.
Solution Approach 2:
The patent segments the data handling process into distinct components: small metadata fields remain in the request object, while large binary data objects are extracted, serialized separately, and stored independently. This segmentation allows differential optimization of transfer and storage operations.
2Productivity
If large data objects are serialized and transmitted between microservices, then data transfer capability increases, but serialization time and network consumption increase
Solution Approach 1:
The patent replaces the mechanical process of serializing large binary data objects with a reference-based system. Instead of converting the entire data object to a serializable format, the system stores the object in binary format in object storage and transmits only a reference identifier, dramatically reducing serialization time.
3Speed
If queryable fields are extracted from large data objects, then data retrieval efficiency improves, but processing complexity increases
Solution Approach 1:
The patent extracts only the queryable fields from the large data object and stores them as separate metadata in the request object. This allows the service to perform filtering, sorting, and searching operations on these extracted fields without processing the entire large data object, improving retrieval efficiency while keeping processing complexity manageable through selective extraction.
Data Source
AI summary
In one aspect, an example methodology implementing the disclosed techniques includes, by a computing device, receiving a request to write a first data object to an object database. The method also includes, responsive to a determination that the first data object is a large data object, by the computing device, serializing the first data object, compressing the serialized first data object into a format that can be stored in the object database, and saving the compressed serialized first data object within the object database. The method may further include, by the computing device, extracting one or more fields which are queryable from the first data object and saving the one or more queryable fields with the compressed serialized first data object within the object database.