Unstructured Data Fusion via Linear Pipeline Framework
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data analytics platforms face challenges in handling voluminous, diverse, and fast-paced data sets, requiring mechanisms to fuse and extract value from unstructured data types like images, videos, and text, while reducing cognitive overhead and increasing speed and flexibility.
Innovation Solution
A data analytics platform employing a linear pipeline processing framework with an abstracted query language that defines a data fusion pipeline assembly mechanism, eliminating the need for explicit data location, context, or normalization, and leveraging concurrency for high throughput, using operator groups like storage, locality, selection, and renderer to process disparate data types.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional data processing systems are used to handle big data, then data processing can be performed, but the systems cannot adequately handle voluminous and complex data sets
Solution Approach 1:
The patent segments the data processing system into a linear pipeline with distinct operator groups (storage, locality, selection, renderer) that process data in discrete stages. Each operator handles specific tasks independently, allowing the system to scale and handle complex data sets without overwhelming complexity by breaking down the processing into manageable segments.
Solution Approach 2:
The patent introduces an abstracted query language dimension that operates above the physical data storage and processing layers. This abstraction layer allows users to define data fusion pipelines without explicitly specifying data locations, contexts, or normalization details, thereby simplifying the interface while maintaining underlying system complexity for handling voluminous data.
2Ease of operation
If normalization is applied to convert all data to text data, then data can be processed uniformly, but normalization becomes the dominating factor limiting data ingest rate
Solution Approach 1:
The patent performs normalization preliminarily during the data ingestion phase rather than as a bottleneck step later in processing. The storage operator group normalizes diverse data types (images, videos, text, binaries) into a unified format structure as data enters the pipeline, allowing high-speed ingestion to continue while normalization occurs in parallel, thus maintaining both uniformity and high ingest rates.
Solution Approach 2:
The patent introduces an intermediary normalized data structure that serves as a universal format between diverse input data types and the processing operators. This intermediary representation allows different data types to be converted to a common internal format without limiting the ingest rate, as the conversion occurs through efficient operator transformations rather than as a serial bottleneck.
3Measurement precision
If explicit definition of data location, context, extraction, or normalization is required, then data processing can be precise, but cognitive overhead on human operators increases
Solution Approach 1:
The patent implements self-service mechanisms where the system automatically determines data locations, contexts, and normalization requirements without requiring explicit user specification. The operator groups autonomously retrieve data from appropriate storage locations, interpret data contexts, and apply necessary normalizations based on metadata and data characteristics, thereby maintaining processing precision while eliminating cognitive overhead for these detailed decisions.
Solution Approach 2:
The patent introduces metadata and operator intermediaries that bridge the gap between high-level user queries and low-level data processing details. The abstracted query language serves as an intermediary that translates user intent into precise processing operations without requiring users to specify implementation details like data locations or normalization rules, thus maintaining precision while reducing cognitive load.
4Productivity
If concurrency is leveraged for high throughput, then data processing speed increases, but system complexity increases
Solution Approach 1:
The patent segments the concurrent processing system into independent operator groups that can execute in parallel without complex coordination requirements. Each operator (storage, locality, selection, renderer) operates as an independent concurrent task processing specific data transformations, allowing high throughput through parallel execution while maintaining manageable complexity through clear separation of concerns and well-defined operator interfaces.
Data Source
AI summary
The disclosure relates to a data analytics platform in which a linear pipeline processing framework may use an abstracted query language to define a data fusion pipeline assembly mechanism. More particularly, the linear pipeline processing framework may include various operator groups that work in conjunction to organize data entries that can have substantially disparate data types (e.g., text, binary, video, audio, etc.) into a single normalized stream such that one or more processing modules may perform type-specific data processing and feature extraction, normalize an output into a single stream, and finally render the different data types as a fused output.