Template-Based Processing of Heterogeneous User Content

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

The processing of heterogeneous user-generated content from online professional networks and external sources is hindered by the presence of unstructured, semi-structured, and diverse data formats, making manual and automated analysis inefficient due to the large volume and variability in content types, such as posts, updates, comments, and feedback, which are difficult to read and analyze in a timely manner.

Innovation Solution

A system and method for template-based processing of user-generated content, where a content-processing system generates standardized records from text-based representations, using custom configurations for each data source to convert original fields into required fields, perform text mining, and facilitate analysis, enabling scalable and flexible processing of content items across different data sources.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If manual analysis methods are used to process heterogeneous user-generated content, then analysis accuracy may be maintained, but processing efficiency and productivity deteriorate due to large volume and diversity of content

Engineering Contradiction:

Improveprocessing efficiencyVSAvoiddata format diversity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent segments heterogeneous content into standardized fields (e.g., user information, content information, interaction information) that can be processed independently. Each field type is handled by specific extraction rules, allowing parallel processing of different content types while maintaining overall system efficiency.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a standardized record structure as an intermediary between diverse data sources and analysis systems. This intermediate format统一的 various heterogeneous content types, enabling efficient automated processing without losing important information.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If automated text analytics are applied to heterogeneous content, then processing speed and productivity improve, but measurement precision and analysis quality worsen due to data variability

Engineering Contradiction:

Improveprocessing speedVSAvoidanalysis quality

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The patent applies different extraction and processing rules to different field types based on their specific characteristics. For example, user information fields use different validation and normalization rules compared to content text fields, ensuring each type is processed with appropriate precision while maintaining overall automation.

Inventive Principle:

Principle #3Local quality

3Adaptability or versatility

If custom processing scripts are created for each data source, then adaptability to specific sources improves, but device complexity and maintenance burden increase

Engineering Contradiction:

Improvedata source adaptabilityVSAvoidscript complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent creates a universal processing framework that can handle multiple data source types through a common record structure and standardized field definitions. The system accommodates different content types (posts, updates, comments, feedback) using the same core processing logic, reducing the need for separate custom scripts.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The patent implements dynamic field mapping that automatically adapts to different data sources by configuring which fields to extract and how to transform them. This allows the system to handle new data sources and content types without requiring manual script modifications, as the extraction rules can be dynamically adjusted based on content characteristics.

Inventive Principle:

Principle #15Dynamics

4Loss of information

If comprehensive text mining is performed on all content fields, then information completeness improves, but processing time and energy consumption increase

Engineering Contradiction:

Improveinformation completenessVSAvoidprocessing energy

Core Design Contradiction:

Loss of informationVSUse of energy by moving object

Solution Approach 1:

The patent applies text mining and analysis selectively to fields that are most valuable for the specific analysis objectives. Rather than processing all fields uniformly, the system identifies and prioritizes key fields for deep analysis while using lighter processing for less critical fields, reducing overall energy consumption while maintaining essential information completeness.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS10146881B2Scalable processing of heterogeneous user-generated content

Publication Date: 2018.12.04 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

The disclosed embodiments provide a system for processing data. During operation, the system obtains a first configuration for processing a first set of content items from a first data source and a second configuration for processing a second set of content items from a second data source. For each content item in the first set of content items, the system uses mappings from the first configuration to transform original fields from the content item into required fields in a record representing the content item. Next, the system generates, from the required fields, a document key for the content item. The system also performs deduplication of multiple records with the document key and stores a single record with the document key. Finally, the system uses the second configuration to generate, from the second set of content items, a set of records independently of processing the first set of content items.