Multi-Source Heterogeneous Data Processing via Pre-Configured Conversion Library

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data processing systems face challenges in aggregating field data from different data sources due to inconsistent field definitions and incomplete data, making it difficult to use and integrate multi-source heterogeneous data effectively.

Innovation Solution

A method and device that convert field data from various data sources into unified multi-source heterogeneous standard data by determining target standard attribute fields using a pre-configured conversion field library, performing deduplication processes, and synthesizing data across sources, thereby improving data integrity and usability in aggregation.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If field data from different data sources is directly aggregated, then data aggregation can be performed quickly, but data inconsistency and incompleteness occur due to different field definitions

Engineering Contradiction:

Improvedata aggregation efficiencyVSAvoiddata consistency

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent applies preliminary action by pre-configuring a conversion field library that maps heterogeneous field definitions from multiple data sources to a unified standard attribute field system. This preparation work is done before data aggregation, enabling automatic field conversion during the aggregation process without sacrificing efficiency. The conversion rules are established in advance, so when data from different sources is aggregated, the system can automatically transform fields according to the pre-defined mappings, ensuring data consistency while maintaining aggregation speed.

Inventive Principle:

Principle #10Preliminary action

2Device complexity

If field data from multiple data sources is aggregated without conversion, then aggregation process is simple, but data integrity and usability are poor due to heterogeneous field definitions

Engineering Contradiction:

Improveaggregation process complexityVSAvoiddata integrity

Core Design Contradiction:

Device complexityVSLoss of information

Solution Approach 1:

The patent introduces an intermediary mechanism - the conversion field library - that acts as a mediator between heterogeneous data sources and the unified data aggregation system. This library contains mapping relationships between various data source field definitions and standard attribute fields. When data is aggregated, the conversion field library automatically translates fields from different sources into a unified format, preserving data integrity without requiring complex manual conversion processes. This intermediary layer simplifies the aggregation process while ensuring data quality.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Reliability

If data fields from different data sources are converted to unified standard fields, then data consistency is improved, but processing time and computational resources increase

Engineering Contradiction:

Improvedata consistencyVSAvoiddata processing time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent reduces processing time by performing the conversion logic preparation in advance. The conversion field library is pre-configured with all necessary mapping relationships between heterogeneous field definitions and standard attribute fields. During actual data aggregation, the system only needs to look up and apply these pre-defined conversion rules rather than performing complex real-time analysis, significantly reducing processing time while maintaining data consistency.

Inventive Principle:

Principle #10Preliminary action

4Adaptability or versatility

If a pre-configured conversion field library is used to map fields to standard attributes, then data standardization is achieved, but system complexity increases due to library configuration and maintenance

Engineering Contradiction:

Improvedata standardization capabilityVSAvoidsystem configuration complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent applies universality by designing the conversion field library with a standardized structure that can handle multiple data sources with different field definitions through a unified mapping approach. The library is configured once with general conversion rules that can be applied across various data aggregation scenarios. This universal design allows the system to adapt to different data sources without requiring separate configuration for each source, reducing the complexity burden despite the enhanced standardization capability.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS11170022B1Method and device for processing multi-source heterogeneous data

Publication Date: 2021.11.09 SHANGHAI ICEKREDIT INC

AI summary

Disclosed are a method and a device for processing multi-source heterogeneous data. The data source to be processed of multi-source heterogeneous data and the field data of the field to be converted under each data source to be processed are determined, then the target standard attribute field of the field to be converted under each data source to be processed in the target data dimension is determined from a pre-configured conversion field library. Then, the fields to be converted under each data source to be processed are converted into corresponding target standard attribute fields, to obtain the field data of the target standard attribute field under each data source to be processed, thereby synthesizing the multi-source heterogeneous standard data of the target data dimension.