Remote Promise Updates for Low-Latency Streaming Pipelines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional approaches to streaming data pipelines across multiple physical machines incur significant data throughput costs due to the transfer of raw data across machine boundaries, leading to inefficiencies in data movement and performance.
Innovation Solution
Implementing remote descriptors and streaming batches to enable RDMA transport of serialized objects, where remote descriptors uniquely identify the location of data in a data center, allowing on-demand access and reducing the need for bulk data transfers, and streaming batches process remote descriptors rather than actual data to minimize data movement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If raw data is transferred in bulk across machine boundaries in streaming data pipelines, then data can be moved between physical machines, but data throughput is significantly reduced and latency increases
Solution Approach 1:
The patent segments data transfer operations into two distinct phases: (1) transfer of remote descriptor objects that contain metadata and access information, and (2) on-demand transfer of actual data contents. This segmentation allows the system to first exchange lightweight descriptor objects to establish data access paths, then retrieve actual data only when needed, significantly reducing unnecessary data movement and improving throughput while lowering latency
Solution Approach 2:
The patent implements preliminary action by transferring remote descriptor objects before actual data transfer. These descriptors contain pre-computed metadata including data location, size, and access parameters. By performing this preliminary descriptor exchange, the system prepares the groundwork for efficient subsequent data retrieval without immediately transferring bulk data, thereby reducing initial latency and enabling optimized data paths
2Loss of energy
If remote descriptors are used to enable on-demand data access, then data transfer costs are reduced, but the system complexity increases due to descriptor management
Solution Approach 1:
The patent introduces remote descriptor objects as intermediary entities that mediate between data producers and consumers across machine boundaries. These descriptors encapsulate all necessary metadata (location, size, access parameters) and serve as lightweight proxies for actual data. This intermediary layer simplifies the overall system by providing a standardized interface for on-demand data access, reducing the need for complex point-to-point data transfer logic while lowering energy costs through selective data movement
3Productivity
If streaming batches process remote descriptors instead of actual data, then data movement is minimized, but the ability to process complete data sets is compromised
Solution Approach 1:
The patent implements dynamic processing by allowing streaming batch operations to work with remote descriptors for metadata-level operations (filtering, aggregation, routing decisions) while maintaining the capability to retrieve and process complete data sets when required. The system dynamically adapts between descriptor-only processing and full data processing based on operational needs, achieving both high efficiency for routine operations and complete reliability when full data processing is necessary
Data Source
AI summary
Technologies for enabling downstream components to update upstream states in streaming pipelines are described. One method of a first computing device receives a remote promise object assigned to a first serialized object from a second computing device in the data center over a network fabric. The remote promise object uniquely identifies a first contiguous block of the first serialized object stored in a memory associated with the second computing device. The method obtains contents of the first contiguous block and sends contents of a second serialized object back to the second computing device to release the remote promise object.