Executor Grouping and Input Batching for Multi-Core Data Stream Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data stream processing systems face bottlenecks such as instruction cache misses, instruction queue full stalls, and costly memory accesses across CPU sockets, limiting scalability and execution times, especially on multi-core and multi-socket processors due to frequent context switching and non-uniform memory access.
Innovation Solution
The solution involves executor grouping to reduce cross-socket communication, input batching to minimize thread context switches, and increasing TLB page size to improve instruction cache performance and balance memory access wait times across cores, thereby reducing front-end stalls and memory access penalties.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data stream processing systems are run on multi-core processors to increase processing capacity, then computational throughput is improved, but front-end stalls and instruction cache misses increase significantly
Solution Approach 1:
The system segments the data stream processing workload into discrete batches that can be processed independently. By dividing the continuous data stream into manageable chunks, the system reduces instruction cache misses and front-end stalls while maintaining parallel processing capabilities across multiple cores, thereby resolving the contradiction between processing capacity and execution time
Solution Approach 2:
The system performs preliminary actions by pre-fetching and batching data before processing. Input batching prepares data in advance in a format optimized for parallel processing, reducing the need for frequent context switching and instruction cache accesses during execution, thus improving both throughput and reducing stalls
2Productivity
If the system scales to multiple CPU sockets to increase processing capacity, then computational power is improved, but memory access latency increases due to NUMA architecture
Solution Approach 1:
The system applies local quality by binding executors to specific CPU sockets and allocating memory locally to each socket. This ensures that data processing occurs on the same socket where the data resides in memory, eliminating costly cross-socket memory accesses and reducing latency in NUMA architectures while maintaining high processing capacity
Solution Approach 2:
The system introduces an intermediary layer (executor grouping mechanism) that mediates between the data stream processing tasks and the underlying NUMA architecture. This intermediary optimizes task distribution across sockets and ensures data locality, reducing memory access latency while preserving processing capacity
3Productivity
If frequent context switching is performed to utilize multiple threads, then parallel processing capability is improved, but instruction cache performance deteriorates
Solution Approach 1:
The system segments the parallel processing workload into distinct batched units, each processed by dedicated executor threads. This segmentation reduces the frequency of context switching between threads while maintaining parallel processing capability, as each thread can process multiple batches sequentially without interfering with others' instruction cache
Solution Approach 2:
The system ensures continuity of useful action by designing executor threads that process multiple input batches continuously without context switching. Each executor maintains its instruction cache warm by processing sequential batches, eliminating the performance penalty of frequent context switching while preserving parallel processing throughput
Data Source
AI summary
Described is a system, a method, and a computer-implemented apparatus for increasing computational efficiency and capacity of data stream processing systems. In one embodiment, executor grouping reduces cross-socket communication in a Non-Uniform Memory Access (NUMA) system. In another embodiment, input batching reduces thread context switches which improves instruction cache performance.