Data Compaction for Distributed Processing Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Customer relationship management (CRM) software faces inefficiencies in processing large data volumes, particularly due to the 'small files problem' caused by micro-batch processing, leading to performance degradation and increased I/O operations, which burdens memory utilization and hampers real-time data processing.
Innovation Solution
A mechanism for compacting data into customized file sizes using metadata analysis to optimize processing by distributed computing resources, balancing file size to prevent excessive execution time or unnecessary I/O overhead, allowing for parallel processing and efficient resource management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If micro-batch processing is used to handle data streams, then real-time data processing capability is improved, but the small files problem occurs leading to increased I/O operations and performance degradation
Solution Approach 1:
The patent merges multiple small data files into larger consolidated files through a file management system that tracks file sizes and combines files below a threshold size. This reduces the total number of files and I/O operations while preserving real-time processing capabilities through selective merging operations.
Solution Approach 2:
The system dynamically adjusts file size parameters by monitoring file growth and triggering merge operations when files exceed or fall below predetermined size thresholds. This parameter-based control optimizes the balance between file count and file size to improve processing performance.
2Productivity
If file size is increased to reduce I/O operations, then data processing performance is improved, but memory utilization becomes burdened
Solution Approach 1:
The patent implements dynamic file size management where the system continuously monitors file sizes and adapts merging operations based on current conditions. Files are merged dynamically when size thresholds are reached, creating an optimal balance between file size for performance and manageable sizes for memory usage.
Solution Approach 2:
The data storage is segmented into multiple files with controlled size ranges rather than single large files. The system maintains a distribution of file sizes within optimal ranges, preventing both the small files problem and excessive memory burden from single large files.
3Reliability
If more I/O operations are performed to handle small files, then data access completeness is improved, but system resource consumption increases
Solution Approach 1:
By merging small files into larger consolidated files, the system reduces the total number of I/O operations required to access the same data volume. The merge operation preserves data completeness while reducing the frequency of I/O calls to the storage system.
Data Source
AI summary
Described is a system and method for compacting data into customized (e.g. optimal) file sizes for processing by computing resources. The mechanism may leverage various computing resources such as a cluster computing frameworks combined with a stream processing platform to efficiently process the activity data. For example, activity data of an organization may be processed by a set of jobs (or sub-jobs) as part of a data stream by a set of distributed computing resources. In order to efficiently process such data, the mechanism may compact the data into customized (e.g. optimal) file sizes. For example, the customized file sizes may provide an optimal (or near optimal) amount of data to be processed by each job, for example, to improve performance.