Storage System Data Partitioning for Machine Learning Bandwidth Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern electronic systems face challenges in processing large amounts of data, leading to reduced performance and functionality due to increased system bandwidth consumption, access conflicts, and resource consumption, which existing technologies have not adequately addressed.
Innovation Solution
An electronic system with a storage interface and a storage control unit that implements a preprocessing block for partitioning data and a learning block for distributing machine learning processes, utilizing in-storage computing to offload functions and reduce input/output bandwidth, thereby improving system performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If large amounts of data are processed to improve device or system performance and functionality, then performance and functionality are improved, but system bandwidth consumption increases and system access conflicts are introduced
Solution Approach 1:
The patent segments data processing by implementing a preprocessing block that partitions data into different categories (e.g., hot data, cold data, frequently accessed data, rarely accessed data) before machine learning processing. This segmentation allows the system to prioritize processing of important data while reducing overall bandwidth consumption by processing only necessary portions of large data sets.
Solution Approach 2:
The preprocessing block performs preliminary actions by partitioning and categorizing data before it reaches the machine learning processing stage. This preliminary organization reduces the burden on the main processing system by pre-sorting data according to access patterns and importance, thereby reducing system bandwidth consumption during actual processing.
2Productivity
If large amounts of data are processed to improve device or system performance and functionality, then performance and functionality are improved, but system access conflicts are introduced
Solution Approach 1:
The system segments data access by creating different processing queues or channels based on data categories. Frequently accessed data is handled separately from rarely accessed data, reducing contention for shared resources and minimizing system access conflicts while maintaining high performance for critical operations.
Solution Approach 2:
The preprocessing block acts as an intermediary between data storage and machine learning processing. It mediates access conflicts by pre-organizing data and determining processing priorities before data reaches the main processing system, thereby reducing direct contention for system resources.
3Productivity
If large amounts of data are processed to improve device or system performance and functionality, then performance and functionality are improved, but system resources are consumed
Solution Approach 1:
The system applies partial action by processing only the most relevant portions of large data sets based on the preprocessing categorization. Instead of processing all data uniformly, the machine learning block focuses on high-priority data segments identified by the preprocessing block, reducing overall resource consumption while maintaining system performance.
Solution Approach 2:
The preprocessing block applies local quality by assigning different processing priorities to different data segments based on their characteristics. Frequently accessed or important data receives higher priority processing with more resources, while less critical data is processed with fewer resources or lower priority, optimizing overall resource utilization.
Data Source
AI summary
An electronic system includes: a storage interface configured to receive system information; a storage control unit, coupled to the storage interface, configured to implement a preprocessing block for partitioning data based on the system information; and a learning block for processing partial data of the data for distributing machine learning processes.