Distributed Data Processing Latency Reduction via Primary Storage Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed processing systems suffer from latency issues, limiting their use in environments requiring fast data processing, such as certain signal processing scenarios.
Innovation Solution
A method and system that divide a data set into portions, each stored in a primary storage of a dedicated node, allowing processors to access data directly without additional storage devices, enabling efficient processing and result generation through modules configured for parallel processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored in distributed storage systems, then data processing capacity is improved, but access latency increases
Solution Approach 1:
The patent segments data into distinct portions and assigns each portion to a specific node's primary storage. This segmentation allows processors to access only the specific data portions they need without waiting for other data, thereby reducing access latency while maintaining distributed processing capacity.
Solution Approach 2:
The patent introduces primary storage as an intermediary layer between processors and distributed storage. This intermediary (primary storage) acts as a buffer that processors can access directly and quickly, eliminating the latency of direct distributed storage access while maintaining the benefits of distributed data storage.
2Speed
If data is stored in primary storage of dedicated nodes, then data access speed is improved, but system complexity increases
Solution Approach 1:
The patent applies local quality by giving each node its own primary storage specifically dedicated to storing data portions for that node's processors. This local storage configuration optimizes data access speed for each node while keeping the overall system manageable through standardized node designs.
Data Source
AI summary
In one embodiment, a first portion and a second portion of a data set are identified. The first portion of the data set is sent to a first node that stores it in a first primary storage. The first primary storage is configured such that a first processor of the first node accesses information stored in the first primary storage without accessing another storage device. The second portion of the data set is sent to a second node that stores it in a second primary storage. The second primary storage is configured such that a second processor of the second node accesses information stored in the second primary storage without accessing another storage device. The first node generates a first set of results by processing the first portion of the data set. The second node generates a second set of results by processing the second portion of the data set.