Data Structure Marshalling Unit for Non-Linear Traversal

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data processing workloads with non-linear address access patterns in data structures, such as sparse tensors, experience poor performance due to frontend and backend stalls caused by branch misprediction and data-dependent control flow.

Innovation Solution

A data structure marshalling unit with data structure traversal circuitry operating according to a dataflow architecture, comprising multiple layers of traversal circuit units that load elements from data storage circuitry based on programmable iteration ranges, where downstream layers' iteration ranges depend on elements loaded by upstream layers, and output interface circuitry marshals elements into vectors for processing.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If data structure traversal is performed using traditional software on general purpose processors, then the implementation is flexible and adaptable, but performance deteriorates due to branch misprediction and frontend stalls

Engineering Contradiction:

Improvedata structure traversal performanceVSAvoidfrontend stalls and branch misprediction penalties

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The patent replaces the mechanical control flow system of traditional processors (which uses branch instructions and branch prediction) with a dataflow architecture where traversal is driven by data availability rather than sequential control instructions. This eliminates branch misprediction penalties by substituting the control-mechanism with a data-driven mechanism where traversal proceeds when required data elements are available in the buffer, not when a branch instruction is executed.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent segments the data structure traversal function into separate hardware components: a buffer for storing data elements, traversal circuitry for processing, and interface circuitry for data movement. This segmentation allows parallel operation of multiple traversal units and eliminates the sequential execution bottleneck of traditional software implementations, directly improving productivity while reducing time loss through concurrent processing.

Inventive Principle:

Principle #1Segmentation

2Productivity

If branch prediction is used to handle data-dependent control flow, then execution continues without stalls, but performance deteriorates due to misprediction penalties

Engineering Contradiction:

Improveexecution continuityVSAvoidbranch prediction accuracy

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent substitutes the branch prediction mechanism with a dataflow-based control system. Instead of predicting whether a branch will be taken and speculatively executing instructions, the system waits for data elements to be loaded into the buffer and then automatically proceeds with traversal when the data is available. This eliminates misprediction penalties entirely by replacing the predictive control mechanism with a reactive data-driven mechanism.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Adaptability or versatility

If data structures are accessed with non-linear address patterns, then complex data relationships can be represented, but performance deteriorates due to poor memory access efficiency

Engineering Contradiction:

Improvedata structure representation capabilityVSAvoidmemory access efficiency

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The patent implements preliminary action by loading data elements into a buffer before they are needed for traversal operations. The interface circuitry proactively moves data from storage to the buffer based on anticipated traversal requirements, rather than waiting for each access request. This preliminary data preparation improves memory access efficiency by batch-processing data movements and reducing the overhead of individual non-linear accesses.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces a buffer as an intermediary between storage and the traversal processing unit. This buffer absorbs the non-linear access patterns by storing data elements in advance, allowing the traversal circuitry to operate with linear, sequential access to buffered data. The intermediary buffer decouples the complex non-linear access requirements from the processing unit, maintaining both data structure versatility and memory access efficiency.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS20250077499A1Data structure marshalling unit

Publication Date: 2025.03.06 ARM LTD

AI summary

A data structure marshalling unit for a processor comprises data structure traversal circuitry to perform data structure traversal processing according to a dataflow architecture. The data structure traversal circuitry comprises two or more layers of traversal circuit units, each layer comprising two or more parallel lanes of traversal circuit units. Each traversal circuit unit triggers loading, according to a programmable iteration range, of at least one stream of elements of at least one data structure from data storage circuitry. For at least one programmable setting for the data structure traversal circuitry, the programmable iteration range for a given traversal circuit unit in a downstream layer is dependent on one or more elements of the at least one stream of elements loaded by at least one traversal circuit unit in an upstream layer. Output interface circuitry outputs to the data storage circuitry at least one vector of elements loaded by respective traversal circuit units in a given active layer of the data structure traversal circuitry.