Front-End Interface for Big Data Pre-Processing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current computer systems face performance bottlenecks and high energy consumption when processing large datasets due to the dominant factor of data movement to the CPU for pre-processing and analysis in big data applications, leading to increased power consumption and thermal design power limitations.

Innovation Solution

A front-end interface with local processors determines whether data is single-access or multiple-access, routing single-access data for pre-processing and reducing energy consumption by avoiding transfers to the CPU, thereby overcoming processing bottlenecks and lowering energy consumption.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If data is transferred to CPU for pre-processing and analysis, then processing capability is improved, but energy consumption increases

Engineering Contradiction:

Improveprocessing capabilityVSAvoidenergy consumption

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The system segments data into single-access and multiple-access data, routing single-access data through a separate processing path (front-end interface with local processors) that bypasses the CPU, thereby reducing CPU energy consumption while maintaining processing capability for data that requires it

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

A front-end interface with local processors is introduced as an intermediary between storage and CPU, performing pre-processing of single-access data locally before results are transferred to CPU, thereby reducing the energy-consuming data movement to CPU

Inventive Principle:

Principle #24Intermediary (Mediator)

2Ease of operation

If data is moved to main memory for processing, then data accessibility is improved, but power consumption increases

Engineering Contradiction:

Improvedata accessibilityVSAvoidpower consumption

Core Design Contradiction:

Ease of operationVSUse of energy by stationary object

Solution Approach 1:

The system segments data access patterns and routes single-access data through a path that avoids main memory transfers, keeping such data processed locally at the front-end interface, thereby reducing power consumption while maintaining accessibility for multiple-access data

Inventive Principle:

Principle #1Segmentation

3Reliability

If all data is processed by CPU, then processing thoroughness is improved, but device complexity increases

Engineering Contradiction:

Improveprocessing thoroughnessVSAvoidsystem complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system segments processing tasks between front-end interface local processors and CPU based on data access patterns, ensuring single-access data is handled efficiently locally while multiple-access data receives thorough CPU processing, thereby maintaining processing thoroughness without excessive complexity

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The front-end interface with local processors provides multi-functionality by handling both pre-processing of single-access data and coordination with CPU for multiple-access data, reducing the burden on CPU while maintaining comprehensive processing capability

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS10417005B2Multi-multidimensional computer architecture for big data applications

Publication Date: 2019.09.17 HUAWEI TECH CO LTD

AI summary

A data processing apparatus is provided comprising a front-end interface electronically coupled to a main processor. The front-end interface is configured to receive data stored in a repository, in particular an external storage and/or a network, determine whether the data is a single-access data or a multiple-access data by analyzing an access parameter designating the data, route the multiple-access data for processing by the main processor, and route the single-access data for pre-processing by the front-end interface and routing results of the pre-processing to the main processor.