Logical Partition Filtering for Distributed Streaming Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed streaming systems face limitations in the number of partitions they can support, leading to underutilization and reduced capacity, as customers often create more partitions than needed, resulting in inefficiencies in processing and storage.
Innovation Solution
The system allows customers to create multiple logical partitions within a physical partition, enabling each physical cluster to handle a larger number of partitions, thereby increasing throughput and storage capacity without physical footprint expansion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If physical partitions are increased to support more customer partitions, then the number of supported partitions increases, but the physical footprint and resource consumption increase
Solution Approach 1:
The patent segments a single physical partition into multiple logical partitions, allowing one physical partition to serve multiple customer partitions. This is achieved by implementing a partition service that virtualizes the physical partition resource, creating multiple logical views of the same physical storage space, thereby increasing the number of supported partitions without expanding physical footprint.
Solution Approach 2:
The physical partition is designed to perform multiple functions by serving as the backend storage for multiple logical partitions simultaneously. The partition service enables a single physical partition to fulfill the requirements of numerous customer partitions through logical abstraction and virtualization, making the physical resource universal and multi-functional.
2Quantity of substance
If physical clusters are expanded to handle larger data volumes, then storage capacity increases, but system complexity and resource requirements increase
Solution Approach 1:
The patent introduces a logical dimension to the partition hierarchy, creating logical partitions within physical partitions. This dimensional transformation allows the system to accommodate larger data volumes and more partitions by utilizing logical abstraction rather than physically expanding the cluster, thereby increasing storage capacity while minimizing the increase in system complexity.
3Loss of energy
If the number of physical partitions is limited, then resource consumption is minimized, but customer partition allocation flexibility is reduced
Solution Approach 1:
The patent creates multiple logical copies or views of the same physical partition, allowing customers to allocate numerous partitions without duplicating physical resources. The partition service virtualizes the physical partition into multiple logical partitions that can be allocated to different customers, maintaining resource efficiency while providing allocation flexibility.
Data Source
AI summary
Techniques for storing and processing large volumes of data by increasing the number of partitions that customers can allocate and use for their streams while minimizing the physical footprint of the partitions is disclosed. In certain examples, the stream partitions are created as different logical partitions on a physical partition in a physical storage cluster of a distributed message streaming system. The disclosed system provides an improved process for reading messages stored in different logical partitions of a physical partition by identifying partition chunks in a physical partition that comprise subsets of messages associated with a logical partition and filtering out partition chunks that do not contain messages associated with the logical partition. The disclosed system reduces the amount of the physical partition that needs to be read to retrieve messages stored in different logical partitions thereby improving the overall performance of the distributed message streaming system.