Distributed Object Store Metadata Exposure for Hadoop
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current object store systems primarily focus on storing and retrieving object data but do not efficiently expose object metadata to distributed processing systems, limiting their ability to process large-scale metadata effectively.
Innovation Solution
A distributed object store system is designed to expose object metadata as a collection of records compatible with frameworks like Hadoop, using a distributed key/value store to determine metadata locations and format them into formats such as Apache Avro, Apache Thrift, or JSON for efficient processing by MapReduce and other distributed systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If object store systems store and retrieve object data efficiently, then data storage and retrieval performance is improved, but object metadata exposure to distributed processing systems becomes inefficient
Solution Approach 1:
The patent segments metadata into structured records with specific formats (Apache Avro, Apache Thrift, JSON, etc.) that can be independently processed by distributed systems. Each metadata record is formatted as a discrete unit that can be efficiently distributed and processed across multiple nodes, resolving the contradiction between fast data retrieval and efficient metadata processing.
Solution Approach 2:
The patent introduces an intermediary layer that converts object metadata into distributed processing-compatible formats. This intermediary formatting layer acts as a bridge between the object store system and distributed processing systems like Hadoop, enabling metadata to be exposed and processed efficiently without compromising data storage performance.
2Adaptability or versatility
If object metadata is exposed to distributed processing systems, then data analysis capabilities are improved, but system complexity increases
Solution Approach 1:
The patent implements universal metadata record formats that can be processed by multiple distributed processing systems (Hadoop, Spark, etc.). By adopting standard formats like Apache Avro, Apache Thrift, and JSON, the system achieves multi-functionality and broad compatibility without increasing complexity, as these formats are widely supported across different processing frameworks.
Solution Approach 2:
The patent changes the parameter of metadata representation from proprietary binary formats to standardized text-based or structured formats. This parameter change enables interoperability with distributed processing systems while maintaining manageable system complexity, as the formatting layer uses well-established, widely-supported serialization formats.
3Productivity
If metadata is processed in conjunction with object data, then scalability is improved, but processing overhead increases
Solution Approach 1:
The patent applies preliminary action by pre-formatting metadata into distributed processing-compatible formats during object ingestion. This preliminary formatting eliminates the need for complex real-time conversion during processing, reducing processing overhead while enabling scalable metadata handling across distributed systems.
Solution Approach 2:
The patent creates equipotentiality by ensuring metadata is available in the same standardized formats as object data, allowing both to be processed uniformly by distributed systems. This eliminates the need for separate processing pipelines and reduces overall processing overhead while maintaining scalability.
Data Source
AI summary
A distributed object store can expose object metadata, in addition to object data, to distributed processing systems, such as Hadoop and Apache Spark. The distributed object store may acts as a Hadoop Compatible File System (HCFS), exposing object metadata as a collection of records that can be efficiently processed by MapReduce (MR) and other distributed processing frameworks. Various metadata records formats are supported. Related methods are also described.