Metadata Schema Workflows for Large-Scale Data Processing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing large-scale data processing systems force all data to flow directly between nodes in a computational graph, require users to write specialized code for data handling, lack an independent task orchestrator, and provide inadequate data caching, leading to inefficiencies in processing large datasets.

Innovation Solution

A data processing system utilizing a predefined metadata schema to define input and output metadata for operations, enabling workflows to be authored without specialized code, with data handling functionalities like persisting and caching handled by the system, and allowing data to be efficiently persisted and cached between operations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If users write specialized code for data handling and workflow execution, then processing can be customized and adapted to specific needs, but system complexity and difficulty of operation increase

Engineering Contradiction:

Improveworkflow customizationVSAvoidworkflow authoring difficulty

Core Design Contradiction:

Adaptability or versatilityVSEase of operation

Solution Approach 1:

The patent introduces metadata schemas as an intermediary layer between users and the data processing system. Instead of requiring users to write specialized code, they define workflows using structured metadata that describes data characteristics, operations, and execution parameters. This metadata intermediary handles the complexity of code generation and execution, while users work with intuitive schema-based definitions.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system enables self-service by automatically generating execution code from metadata schemas and autonomously managing workflow execution. The metadata schema definitions are automatically translated into executable code by the system, eliminating the need for users to manually write and maintain specialized processing code.

Inventive Principle:

Principle #25Self-service

2Reliability

If data flows directly between nodes in computational graph without caching, then data freshness is maintained, but processing efficiency decreases due to repeated data retrieval

Engineering Contradiction:

Improvedata freshnessVSAvoidprocessing efficiency

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent implements preliminary action by caching data at intermediate stages during workflow execution. Data that has been processed by certain operations is stored in cache memory, allowing subsequent operations to retrieve previously processed data without re-processing the original source data, thus maintaining efficiency while preserving data integrity through controlled cache invalidation.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system applies local quality by implementing selective caching strategies where different data paths and operations have customized caching behaviors. The metadata schemas define cache validity conditions and invalidation triggers specific to each data transformation, allowing the system to cache data locally at relevant processing stages while maintaining freshness where needed.

Inventive Principle:

Principle #3Local quality

3Measurement precision

If existing systems require exact data and operation matching for caching, then cache accuracy is maintained, but cache hit rate decreases leading to more cache misses

Engineering Contradiction:

Improvecache matching accuracyVSAvoidcache hit rate

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The patent transforms the caching approach by changing parameters from exact matching to schema-based matching. Instead of requiring identical data and operation sequences, the system uses metadata schemas that define structural and semantic equivalence. This allows different but semantically equivalent operations to hit the same cache, dramatically increasing cache hit rates while maintaining accuracy through schema validation.

Inventive Principle:

Principle #35Parameter changes

4Reliability

If users must define data distribution and code execution manually, then processing control is precise, but workflow authoring complexity increases

Engineering Contradiction:

Improveprocessing controlVSAvoidworkflow definition complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The metadata schema acts as an intermediary that captures processing control requirements in a structured, declarative format. Users define data distribution and execution control through schema parameters rather than manual code, reducing complexity while maintaining precise control through the schema's type system and validation rules.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS12450207B2Systems and methods for authorizing workflows from a large-scale dataset using a metadata schema

Publication Date: 2025.10.21 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

Systems and methods for authoring workflows for processing data from a large-scale dataset include defining a metadata schema for the large-scale dataset, and receiving user input defining a workflow as a plurality of operations to be performed on the data. Each of the operations includes input metadata formatted according to the metadata schema. The input metadata describes input data to be processed by the operation and identifying a location for the input data in the data storage system, programmed instructions for performing an atomic operation on the input data to generate output data; and output metadata formatted according to the metadata schema. The output metadata describes the output data and identifying a location for the output data in the data storage system.