Asynchronous Data Block Processing for High Throughput Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data analytics systems face performance bottlenecks and delays when processing large data sets due to inefficient data processing techniques, leading to reduced throughput and quality of information extraction.
Innovation Solution
A computer-implemented data analytics system that asynchronously processes data blocks using multiple worker threads in parallel, allowing for speculative processing of data records that span adjacent blocks, with correction mechanisms to ensure accuracy and maintain high throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data records are processed sequentially in traditional data analytics systems, then processing accuracy is maintained, but throughput is reduced to approximately 0.1 GB/sec
Solution Approach 1:
The patent segments data into fixed-size blocks and assigns different blocks to multiple worker threads for parallel processing. Each worker thread independently processes assigned blocks, enabling concurrent data processing across multiple threads while maintaining processing accuracy through block-level segmentation.
Solution Approach 2:
The patent implements preliminary actions by pre-loading data blocks into memory before processing, maintaining a block cache to pre-fetch upcoming blocks, and performing speculative processing of data records that may span block boundaries. These preliminary actions prevent processing delays and maintain high throughput.
2Productivity
If multiple worker threads process data blocks in parallel, then throughput increases to 2.0 GB/sec, but complexity of managing data records spanning adjacent blocks increases
Solution Approach 1:
The patent implements feedback mechanisms where worker threads report processed block ranges and data record completion status to a coordinator. The coordinator tracks which data records have been fully processed and manages the release of processed data to downstream consumers, enabling parallel processing while maintaining coordination through feedback loops.
Solution Approach 2:
The patent introduces intermediary components including a block cache that mediates between data storage and worker threads, and a coordinator that mediates between multiple worker threads and downstream data consumers. These intermediaries simplify the complexity of direct worker-thread coordination by providing buffering and management layers.
3Speed
If data blocks are read sequentially from storage device, then data integrity is ensured, but sustained data transfer rate cannot match data source capacity
Solution Approach 1:
The patent implements preliminary actions by pre-loading data blocks into a block cache from storage devices before they are needed for processing. This allows the system to maintain sustained data transfer rates that match the data source capacity by having data ready in memory before workers need it, while still ensuring data integrity through proper caching protocols.
Solution Approach 2:
The patent transitions from sequential single-threaded processing to parallel multi-threaded processing, adding a dimension of concurrency. Multiple worker threads process different data blocks simultaneously, enabling the system to achieve sustained data transfer rates that match the underlying storage capacity while maintaining data reliability through coordinated processing.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A data analytics system stores a plurality of sequential data blocks in a block cache. Each data block contains one or more data records. Each data record of a data record contains one or more values and a record marker marking the end of the data record. The data analytics system allocates a data block in the block cache to a worker thread. The worker thread parses the data block to identify the one or more data records contained within the data block. The data analytics system determines whether the data block was correctly parsed. Responsive to determining that the data block was correctly parsed, the data analytics system provides the one or more data records contained within the data block to a data analytics workflow.