Recurrent Neural Network for Unordered Input Set Processing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional recurrent neural networks are limited in processing unordered input sets and training on target output sets without specified order, as they rely on ordered sequences, which restricts their ability to generate consistent outputs regardless of input order and requires predefined output order during training.

Innovation Solution

A recurrent neural network system comprising a read neural network, a process neural network, and a write neural network that generates an order-invariant numeric embedding from unordered input sets, allowing the system to produce consistent outputs regardless of input order, and a training system that selects an optimal order for target outputs to train the network effectively.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If conventional recurrent neural networks use ordered sequences for processing, then the network structure and training process are simple and well-defined, but the network cannot process unordered input sets and cannot generate consistent outputs regardless of input order

Engineering Contradiction:

Improveability to process unordered input setsVSAvoidnetwork structure complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The network is divided into three distinct functional modules: a read neural network that processes individual inputs independently, a process neural network that handles permutation-invariant processing, and a write neural network that generates outputs. This segmentation allows each module to specialize in handling unordered data without requiring the entire network to be restructured.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

A permutation-invariant intermediate representation is introduced between the read and write networks. This intermediate layer acts as a mediator that transforms unordered inputs into a form that preserves set properties regardless of input order, enabling the write network to generate consistent outputs without being affected by input permutations.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Adaptability or versatility

If conventional recurrent neural networks require predefined output order during training, then the training process is straightforward with clear target sequences, but the network cannot handle tasks where target output sets have no specified order

Engineering Contradiction:

Improveability to handle unordered target output setsVSAvoidtraining process simplicity

Core Design Contradiction:

Adaptability or versatilityVSEase of operation

Solution Approach 1:

Instead of requiring the network to output sequences in a predefined order, the training approach is inverted: the network generates outputs in arbitrary order, and the training loss function is designed to be permutation-invariant. This allows the network to learn from unordered target sets while maintaining training simplicity through symmetric loss calculations that don't depend on output sequencing.

Inventive Principle:

Principle #13The other way round (Inversion)

3Adaptability or versatility

If the network processes inputs in different orders to account for unordered sets, then the network can handle unordered data, but the processing time and computational resources increase

Engineering Contradiction:

Improvehandling of unordered input setsVSAvoidprocessing time

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The network transforms the problem from processing multiple permutations to processing a single permutation-invariant representation. By changing the parameter space to use set-based operations and permutation-invariant functions, the network achieves unordered data handling without the exponential time cost of evaluating all possible input orderings.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS11829860B2Processing and generating sets using recurrent neural networks

Publication Date: 2023.11.28 GOOGLE LLC

AI summary

In one aspect, this specification describes a recurrent neural network system implemented by one or more computers that is configured to process input sets to generate neural network outputs for each input set. The input set can be a collection of multiple inputs for which the recurrent neural network should generate the same neural network output regardless of the order in which the inputs are arranged in the collection. The recurrent neural network system can include a read neural network, a process neural network, and a write neural network. In another aspect, this specification describes a system implemented as computer programs on one or more computers in one or more locations that is configured to train a recurrent neural network that receives a neural network input and sequentially emits outputs to generate an output sequence for the neural network input.