Verifiable Random Function Sampling for Streaming Data

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing technologies face challenges in providing a scalable, private, and confidential method for data stream owners to share streaming data with potential users, while ensuring the correctness and representativeness of the data sample.

Innovation Solution

The implementation of Verifiable Random Functions (VRFs) at each data source, allowing each source to independently evaluate whether it is selected for the sample using its secret key, and providing inclusion or exclusion proofs to ensure the correctness and randomness of the sampling process.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a data stream owner provides sampled data to potential buyers for verification, then the buyers can verify statistical properties and correctness of the data stream, but the data stream owner cannot ensure that buyers will not exploit the intelligence produced with other providers' data

Engineering Contradiction:

Improvedata correctness verificationVSAvoiddata exploitation by competitors

Core Design Contradiction:

ReliabilityVSObject-generated harmful factors

Solution Approach 1:

A trusted third-party broker is introduced as an intermediary between data stream owners and potential buyers. The broker receives data from multiple owners, performs sampling and verification, and provides aggregated insights to buyers. This mediator structure allows verification of data quality while preventing direct access to raw data that could be exploited by competitors.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If exhaustive enumeration of all useful properties of a data stream is performed, then complete verification of data utility is achieved, but the processing cost and time become prohibitively expensive

Engineering Contradiction:

Improvecomplete data verificationVSAvoidprocessing cost and time

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

Instead of performing exhaustive verification of all possible data properties, the system performs partial verification by sampling a subset of data points and checking a limited set of statistical properties. This partial action approach provides sufficient assurance of data quality while keeping processing costs and time within acceptable limits.

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The system performs preliminary sampling and verification actions before complete data processing. By pre-verifying a representative sample, the system can make informed decisions about data utility without investing resources in exhaustive verification of the entire data stream.

Inventive Principle:

Principle #10Preliminary action

3Measurement precision

If a central authority collects and processes all streaming data to ensure sampling correctness, then verification accuracy is improved, but network bandwidth requirements and processing costs increase significantly

Engineering Contradiction:

Improvesampling verification accuracyVSAvoidnetwork bandwidth and data volume

Core Design Contradiction:

Measurement precisionVSQuantity of substance

Solution Approach 1:

The verification process is segmented and distributed across multiple independent sampling nodes rather than centralized. Each node independently performs sampling and verification on local data subsets, eliminating the need to transmit all raw data to a central authority. This segmentation maintains verification accuracy while dramatically reducing network bandwidth requirements.

Inventive Principle:

Principle #1Segmentation

4Adaptability or versatility

If a data stream owner manipulates the data collection process to skew data towards own benefit, then the owner's competitive position is improved, but the representativeness and correctness of the data sample deteriorates

Engineering Contradiction:

Improvecompetitive advantageVSAvoiddata representativeness

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The system implements feedback mechanisms where sampling results and verification outcomes are publicly reported and can be audited by multiple parties. This transparency creates feedback loops that deter manipulation, as any skewing of data would be detectable and damage the owner's reputation. The feedback mechanism aligns competitive incentives with data integrity.

Inventive Principle:

Principle #23Feedback

Data Source

PatentEP4539389A1Method and system for independent proof-of-correct-sampling of streaming data

Publication Date: 2025.04.16 NOKIA SOLUTIONS & NETWORKS OY

Figure 1
Figure 2
Figure 3

AI summary

Described herein is a network element, a network system, a computer readable medium and a method for determining proof-of-correct-sampling for streaming data in a network. The network element is configured for providing, as a data source, at least a part of a data stream and/or a data item based on the selection of the first network element to be included in a sample of the data stream, wherein the first network element comprises: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the first network element at least to: employ a first function with a secret key, specific to the first network element, to determine whether the first network element is selected to be included in the sample; wherein, if it is determined that the first network element is selected to be included in the sample, provide, to one or more second network elements, at least a part of the data stream and/or data items as part of the sample and an inclusion proof as proof of properties of the data stream and/or data item.