Hash-Based Time-Series Data Validation Structure

Solution Overview

Problem

Existing data validation methods for time-series data become impractical as data volumes increase, consuming excessive memory, processing, and networking resources due to the need for element-by-element comparisons, especially when validating large datasets before transmission to external systems.

Innovation Solution

A system that generates a compressed representation of time-series data using a hash-based structure, allowing for efficient validation by comparing a single hash value to determine data integrity and identifying changes, thereby reducing storage, processing, and networking resources.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If multiple copies of stored data are maintained and compared element-by-element to identify inconsistencies, then data validation accuracy is improved, but memory consumption and processing resources increase significantly

Engineering Contradiction:

Improvedata validation accuracyVSAvoidmemory consumption

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent extracts only the essential validation information from the full data copies by computing hash values. Instead of storing and comparing complete data elements, the system extracts condensed hash representations that capture the essential integrity information while occupying minimal storage space, directly resolving the contradiction between validation accuracy and memory consumption

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent transforms the data representation from full-element format to hash-based format, changing the parameter of data representation. This transformation allows the system to maintain validation capability while dramatically reducing the quantity of stored information, as hash values occupy far less memory than the original data elements they represent

Inventive Principle:

Principle #35Parameter changes

2Reliability

If multiple copies of stored data are maintained and compared element-by-element to identify inconsistencies, then data validation accuracy is improved, but processing resources increase significantly

Engineering Contradiction:

Improvedata validation accuracyVSAvoidprocessing resources

Core Design Contradiction:

ReliabilityVSPower

Solution Approach 1:

The patent extracts only the essential validation information from the full data copies by computing hash values. Instead of storing and comparing complete data elements, the system extracts condensed hash representations that capture the essential integrity information while occupying minimal storage space, directly resolving the contradiction between validation accuracy and memory consumption

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent transforms the data representation from full-element format to hash-based format, changing the parameter of data representation. This transformation allows the system to maintain validation capability while dramatically reducing the quantity of stored information, as hash values occupy far less memory than the original data elements they represent

Inventive Principle:

Principle #35Parameter changes

3Reliability

If element-by-element comparison is performed to validate large datasets, then data integrity is ensured, but networking resources are consumed due to retransmissions

Engineering Contradiction:

Improvedata integrityVSAvoidnetworking resources

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The patent performs validation using hash comparisons before initiating data transmission. By conducting this preliminary check, the system identifies and corrects errors beforehand, preventing the need for retransmissions and thereby conserving networking resources that would otherwise be wasted on transmitting corrupted data

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS11645252B2System and method for efficiently validating time-series data using a hash-based representation of the data

Publication Date: 2023.05.09 BANK OF AMERICA CORP

AI summary

An apparatus includes a memory and a processor. The memory stores a time-series of data sets, and a first version of a data structure generated from the time-series as it existed at a first time. The data structure includes a terminal node that stores a compressed representation of the time-series and a bottom level of nodes, each of which stores a hash of a data set of the time-series. The processor generates a second version of the data structure based on the time-series as it exists at a second time. The processor determines that the data stored by the terminal nodes in the first and second versions do not match. In response, the processor identifies a bottom level node for which the stored data in the first and second versions do not match, and requests validation. The processor receives a response indicating whether an intentional change was made.