Scalable Distributed Real-Time Data Warehousing Engine
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In real-time systems, bulk storage media such as DDR SDRAM chips are inefficiently used due to static memory mapping, leading to high costs, power consumption, and layout challenges, especially as data storage demands increase with evolving wireless standards like 5G, and manual optimization is difficult and time-consuming.
Innovation Solution
A scalable distributed real-time data warehousing system that decouples storage from computation, using a networked bulk storage controller to manage bulk storage media, enabling flexible configurations and algorithmic techniques like compression and error correction to optimize data handling and storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If static memory mapping is used in bulk storage media, then data storage capacity is increased, but memory usage efficiency deteriorates and power consumption increases
Solution Approach 1:
The patent implements dynamic memory mapping that adapts to actual data access patterns. The system uses a data warehouse engine to learn and predict data usage patterns, dynamically remapping memory addresses to optimize access efficiency. This resolves the contradiction by making the memory mapping dynamic rather than static, allowing the system to maintain high storage capacity while improving memory usage efficiency based on actual workload characteristics.
Solution Approach 2:
The system changes the parameter of memory mapping from static to dynamic by introducing a data warehouse engine that continuously learns data access patterns. This parameter change enables the system to adapt memory allocation and mapping strategies based on observed usage patterns, thereby improving memory usage efficiency while maintaining the required storage capacity.
2Speed
If manual optimization of memory usage via overlays is used, then real-time performance is improved, but development time and complexity increase
Solution Approach 1:
The patent implements a self-service approach where the data warehouse engine automatically learns data access patterns and performs optimization without manual intervention. The system autonomously generates memory mapping strategies based on observed usage patterns, eliminating the need for manual optimization while maintaining real-time performance. This resolves the contradiction by automating the optimization process.
Solution Approach 2:
The system incorporates feedback mechanisms where the data warehouse engine continuously monitors data access patterns and uses this feedback to dynamically adjust memory mapping strategies. This closed-loop approach enables the system to automatically adapt to changing workload characteristics and maintain optimal real-time performance without requiring manual reconfiguration or complex development efforts.
3Ease of operation
If global memory mapping is used, then data accessibility is improved, but on-SoC memory utilization efficiency deteriorates due to address space gaps
Solution Approach 1:
The patent applies local quality by creating localized memory mappings that are optimized for specific data access patterns rather than using a uniform global mapping. The data warehouse engine learns which data elements are frequently accessed together and creates localized mappings that minimize address space gaps for those specific access patterns. This resolves the contradiction by making memory mapping locally optimized rather than globally uniform.
Solution Approach 2:
The system segments the memory address space into multiple regions, each optimized for specific data types or access patterns. Instead of using a single global memory map, the data warehouse engine divides the address space and applies different mapping strategies to different segments based on their usage characteristics. This segmentation eliminates address space gaps within each segment while maintaining overall data accessibility.
Data Source
AI summary
A computation system-on-a-chip (CSoC) includes a first scalable distributed real-time Data Warehousing (sdrDW) engine and a network interface coupled to the first sdrDW engine, where the network interface is coupled to an interconnect, and where the CSoC is configured to transmit a task request over the interconnect to a first networked bulk storage controller (NBSC) requesting that a task be performed on a bulk storage medium.