Object Storage Streaming With Metadata Remapping for Broker Load
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed streaming platforms like Apache Kafka require significant computing resources due to multiple brokers, which are partitioned by topic, leading to inefficiencies and increased resource consumption.
Innovation Solution
Implement a background platform that separates metadata from content storage, using object storage services like Amazon S3 and DynamoDB, with a metadata plane that remaps and provides metadata to brokers, allowing for centralized storage and efficient retrieval of content.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple brokers are used in a distributed streaming platform, then content can be partitioned and stored across different nodes, but computing resource requirements increase significantly
Solution Approach 1:
The patent extracts the storage function from the broker nodes and places it in object storage services. Brokers no longer store content locally but instead store metadata pointing to content locations in object storage. This separation allows content partitioning across multiple brokers while eliminating the need for each broker to maintain local content copies, thus reducing computing resource requirements.
Solution Approach 2:
The patent introduces object storage services as an intermediary between brokers and content. Instead of brokers directly storing and managing content, they interact with object storage services through standardized interfaces. This intermediary layer handles content storage, retrieval, and management, allowing brokers to focus on streaming operations with reduced resource consumption.
2Quantity of substance
If content is stored in centralized object storage services, then storage scalability is improved, but access speed may be reduced compared to local broker storage
Solution Approach 1:
The patent implements preliminary action by having brokers send metadata about content locations to the object storage service in advance. The object storage service pre-processes and organizes this metadata, creating an optimized access structure. When content needs to be retrieved, the pre-organized metadata enables fast location lookup and retrieval, compensating for the centralized storage architecture.
Solution Approach 2:
The patent changes the parameter of storage architecture from distributed local storage to centralized object storage with metadata-driven access. By transforming how content is located and accessed (through metadata pointers and timestamps rather than local file systems), the system achieves both scalable storage capacity and maintained access performance.
3Productivity
If different object storage services are used for different content types, then storage optimization is improved, but system complexity increases
Solution Approach 1:
The patent makes the object storage service interface universal and multi-functional. A single object storage service handles multiple content types (video, audio, images, etc.) and multiple brokers through a standardized API. The service can optimize storage for different content types internally while presenting a uniform interface to brokers, thus achieving storage efficiency without increasing apparent system complexity.
Solution Approach 2:
The patent uses parameter changes in metadata (such as content type, access patterns, timestamp) to dynamically determine optimal storage locations and retrieval strategies. The object storage service interprets these metadata parameters to automatically route different content types to appropriate storage configurations, eliminating the need for complex manual configuration and reducing system architecture complexity.
Data Source
AI summary
The described technology pertains to a distributed computing environment, specifically implementing a streaming protocol on a scalable object storage service. The technical problem addressed is the high computing resource demand of distributed streaming platforms. The solution involves separating content from location metadata, allowing centralized, scalable storage of metadata. Brokers store content in object storage services based on specific metrics, such as fast read/write times or greater storage capacity. A background platform remaps metadata sequences based on logical timestamps, optimizing data retrieval. This system enhances the efficiency of computing resources by reducing the load on brokers and improving data access speeds. A use can be in cloud computing systems for efficient content streaming and storage management.