Deep-Structured Conditional Random Fields for Sequential Labeling

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Linear-chain conditional random fields require manual construction of features, lacking the ability to automatically generate robust discriminative internal features from raw features, which limits their performance in sequential labeling tasks.

Innovation Solution

A deep-structured conditional random field model is employed, where multiple layers process input signals, with each layer's output used in conjunction with the input to generate probability information, allowing for the automatic learning of discriminative internal representations and improved classification accuracy.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Extent of automation

If manual feature construction is used in linear-chain CRFs, then the model can achieve acceptable performance with simple structure, but the model lacks the ability to automatically generate robust discriminative internal features from raw features

Engineering Contradiction:
Improveautomatic feature generationVSAvoidmodel structure complexity
Core Design Contradiction:
Extent of automationVSDevice complexity

Solution Approach 1:

The patent segments the feature learning process into multiple hierarchical layers. Each layer learns discriminative features at different levels of abstraction from the raw input, breaking down the complex task of automatic feature generation into manageable stages. This allows the model to automatically generate robust features without requiring a single monolithic complex structure.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The patent introduces a temporal dimension by processing sequences of features through recurrent neural network layers. This transforms the static feature representation into dynamic temporal representations, enabling automatic learning of discriminative internal features across time steps while maintaining a structured approach to complexity.

Inventive Principle:
Principle #17Another dimension (Dimensionality change)

2Reliability

If multiple layers are added to automatically learn discriminative features, then feature robustness improves, but computational complexity and training difficulty increase

Engineering Contradiction:
Improvefeature robustnessVSAvoidcomputational complexity
Core Design Contradiction:
ReliabilityVSDevice complexity

Solution Approach 1:

The patent applies preliminary action by pre-training each layer sequentially before combining them. Lower layers are trained first to learn basic discriminative features, then higher layers are trained on top of these learned representations. This staged approach builds robust features incrementally while managing computational complexity through progressive refinement rather than simultaneous optimization.

Inventive Principle:
Principle #10Preliminary action

Solution Approach 2:

The patent introduces intermediary layers that act as mediators between raw input features and final classification decisions. Each intermediary layer transforms and refines features passed from previous layers, creating a pipeline where computational complexity is distributed and managed across multiple intermediate processing stages rather than concentrated in a single complex operation.

Inventive Principle:
Principle #24Intermediary (Mediator)

Data Source

PatentUS8473430B2Deep-structured conditional random fields for sequential labeling and classification
Publication Date: 2013.06.25 MICROSOFT TECHNOLOGY LICENSING LLC
  • US8473430B2 patent drawing
  • US8473430B2 patent drawing
  • US8473430B2 patent drawing

AI summary

Described is a technology by which a deep-structured (multiple layered) conditional random field model is trained and used for classification of sequential data. Sequential data is processed at each layer, from the lowest layer to a final (highest) layer, to output data in the form of conditional probabilities of classes given the sequential input data. Each higher layer inputs the conditional probability data and the sequential data jointly to output further probability data, and so forth, until the final layer which outputs the classification data. Also described is layer-by-layer training, supervised or unsupervised. Unsupervised training may process raw features to minimize average frame-level conditional entropy while maximizing state occupation entropy, or to minimize reconstruction error. Also described is a technique for back-propagation of error information of the final layer to iteratively fine tune the parameters of the lower layers, and joint training, including joint training via subgroups of layers.