Cloud Data Stream Grouping for Batch Processing Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for updating and modifying large files in cloud storage are inefficient, wasting network bandwidth and resources due to the need to download and re-upload entire files, and can lead to system out-of-memory issues, especially when dealing with huge files and unpredictable data volumes, resulting in inefficient data analysis.
Innovation Solution
A system and method for grouping and storing data streams based on types, where data items are uploaded to a cloud storage, retrieved, and grouped into batch data queues by type, then processed and stored in cloud storage files by batch processing units, reducing unnecessary resource usage and optimizing data analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If a memory buffer is used to implement batch process unit to avoid downloading and re-uploading entire files, then network bandwidth is saved, but the data volume and generating time become unpredictable leading to possible system out-of-memory issues
Solution Approach 1:
The patent implements dynamic buffer management where the batch processing unit continuously monitors data queue volume and adjusts buffer allocation accordingly. When data volume exceeds thresholds, the system dynamically scales buffer resources or splits processing into multiple batches, preventing out-of-memory conditions while maintaining batch processing efficiency and saving network bandwidth.
Solution Approach 2:
The patent segments the data processing into multiple batch operations rather than processing all data at once. The batch processing unit divides large data queues into manageable batches, processing them sequentially or in parallel with controlled memory allocation. This segmentation prevents memory overflow while maintaining the bandwidth-saving benefits of batch processing.
2Reliability
If the size of the memory buffer is limited or process time is fixed to avoid out-of-memory issues, then system stability is improved, but data is split across too many cloud storage files
Solution Approach 1:
The patent employs dynamic buffer sizing and adaptive batch processing where the system adjusts buffer capacity and processing intervals based on real-time data characteristics. This allows the system to maintain stability through controlled buffer limits while minimizing file fragmentation by optimizing batch boundaries to group related data together, reducing the total number of cloud storage files.
3Ease of operation
If each event in AWS lambda processes only one data item from the data queue, then processing is simple, but each event creates its own cloud storage file resulting in scattered data
Solution Approach 1:
The patent merges multiple individual data processing events into consolidated batch operations. Instead of each AWS Lambda event creating a separate file, the batch processing unit accumulates multiple data items and processes them together as a group, creating consolidated cloud storage files. This maintains the simplicity of event-driven architecture while improving data organization and reducing file fragmentation.
4Ease of manufacture
If data files are stored in cloud storage without compacting or grouping, then storage is simple, but data analysis tools waste I/O and network resources loading data from each file
Solution Approach 1:
The patent combines scattered data files into consolidated cloud storage files through batch processing. Data items of the same type or from the same source are grouped together and stored in unified files rather than separate files. This maintains storage simplicity while dramatically improving data analysis efficiency by reducing the number of I/O operations and network requests required to load complete datasets.
Data Source
AI summary
A new approach is proposed to support grouping and storing a data stream based on the types of data items in the stream for efficient data batch processing and analysis. First, the data stream is uploaded to a cloud storage, wherein the stream of data includes a plurality of data items of different types generated by and collected from different users and/or devices. The data items are then retrieved, grouped and saved by a preprocessing unit into a plurality of batch data queues, wherein data items in each batch data queue are of the same type. One or more batch processing units are then configured to fetch and batch process data items from the batch data queues and store these data items of the same data type to one or more cloud storage files for further processing and analysis on the cloud storage one batch data queue at a time.