Universal Data Connector Framework for Bulk Ingestion

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Processing and analyzing vast amounts of data from disparate sources with different formatting, time intervals, and structures poses challenges for real-time retrieval and analysis, especially as data sources change over time, requiring efficient and consistent data ingestion and transformation solutions.

Innovation Solution

A framework for ingesting and transforming data from various sources using preconfigured data connectors with configurable parameters, including connection, time interval, and data transformation parameters, which condenses bulk data into measurable metrics for real-time processing and prediction accuracy improvement.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If custom application code is written for each data source to retrieve data, then data retrieval from specific sources is achieved, but device complexity and development time increase significantly

Engineering Contradiction:

Improvedata source compatibilityVSAvoidcustom code requirements

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent implements a universal data connector framework that can connect to multiple different data sources (relational databases, non-relational databases, message repositories, web services) through a common interface and configuration mechanism, eliminating the need for custom application code for each data source type

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The patent uses configurable parameters and connection strings to adapt the universal connector to different data sources. By changing connection parameters (data source type, connection string, query parameters), the same connector infrastructure can retrieve data from diverse sources without code modification

Inventive Principle:

Principle #35Parameter changes

2Quantity of substance

If data is retrieved and stored in original formats from disparate sources, then data completeness is maintained, but storage space consumption and processing complexity increase

Engineering Contradiction:

Improvedata completenessVSAvoidstorage space

Core Design Contradiction:

Quantity of substanceVSVolume of stationary object

Solution Approach 1:

The patent extracts only the necessary data elements and transformation parameters from the original diverse data formats, storing them in a standardized schema. This extraction approach maintains data completeness for analysis purposes while significantly reducing storage requirements by eliminating redundant formatting information

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent segments data into structured components (metrics, dimensions, time series) with a standardized schema. By dividing data into these organized segments, the system maintains data integrity and completeness while enabling efficient storage and retrieval without preserving the original bulky formats

Inventive Principle:

Principle #1Segmentation

3Stability of the object's composition

If data transformation is performed to standardize formats from different sources, then data consistency is improved, but processing time and computational resources increase

Engineering Contradiction:

Improvedata consistencyVSAvoidtransformation processing time

Core Design Contradiction:

Stability of the object's compositionVSLoss of time

Solution Approach 1:

The patent performs data transformation and standardization at the point of ingestion, converting data to the target schema before storage. This preliminary action ensures data consistency is established upfront, avoiding the need for repeated transformation operations during subsequent analysis and reducing overall processing time

Inventive Principle:

Principle #10Preliminary action

4Adaptability or versatility

If the data ingestion framework is made highly adaptable to accommodate new data sources, then system versatility improves, but system complexity increases

Engineering Contradiction:

Improvenew data source integrationVSAvoidframework complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent implements a universal connector framework with standardized interfaces and configuration mechanisms that can accommodate new data sources through configuration rather than code changes. This universality provides high adaptability while maintaining manageable system complexity through consistent design patterns

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS11487756B1Ingesting and transforming bulk data from various data sources

Publication Date: 2022.11.01 TARGET BRANDS INC

AI summary

In some implementations, a method performed by data processing apparatuses includes receiving configuration data for a preconfigured data connector, including connection parameters, time interval parameters, and data transformation parameters. The connection parameters are used to establish a connection to a bulk data source. In response to determining that an amount of time has elapsed that corresponds to the time interval parameters, bulk data is retrieved from the bulk data source for a given time interval, and the retrieved bulk data is transformed in accordance with the data transformation parameters. Based on transforming the retrieved bulk data, a data metric is generated that condenses the retrieved bulk data. A predetermined predicted metric value is received from a prediction data source for the data metric for a time interval that corresponds to the given time interval, and the predicted metric value is stored with the measured metric value.