Data Translation System Using DTML Executor for Heterogeneous Sources

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Traditional data management systems, such as data warehouses, struggle to handle and integrate data from disparate sources, including unstructured data from sensors and social media, leading to performance degradation and security concerns, especially when trying to perform data analytics across various data types.

Innovation Solution

A system and method that translates data from disparate sources into a homogeneous dataset using a database schema, employing a Data-Translate Markup Language (DTML) executor and data adapters to facilitate meaningful information extraction and analytics, while maintaining security.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If data warehouses are used to store structured data, then data management is highly governed and structured, but the system cannot effectively host unstructured data from sources like sensors and social media

Engineering Contradiction:

Improvecapability to host unstructured dataVSAvoiddata management system complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent introduces a Data Lake as an intermediary layer between unstructured data sources and the structured Data Warehouse. The Data Lake receives and stores raw unstructured data from sensors, social media, and other sources without requiring immediate structuring, while the Data Warehouse continues to manage structured data. This mediator architecture allows the system to handle both structured and unstructured data without forcing a complete system redesign.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent segments the data management system into distinct components: a Data Lake for unstructured data, a Data Warehouse for structured data, and integration layers connecting them. This segmentation allows each component to be optimized for its specific data type and purpose, with the Data Lake handling flexible unstructured data ingestion and the Data Warehouse maintaining governed structured data storage.

Inventive Principle:

Principle #1Segmentation

2Adaptability or versatility

If multiple data warehouses are created to analyze different data types, then analysis capability is improved, but time and energy required to set up and maintain them increases considerably

Engineering Contradiction:

Improveanalysis capability for different data typesVSAvoidtime and energy to set up and maintain data warehouses

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The patent creates a universal Data Lake architecture that can handle multiple data types (structured, unstructured, semi-structured) through a single platform. The system uses configurable schemas and transformation rules that can be adapted to different data sources without requiring separate warehouse installations. This multi-functional approach allows organizations to analyze diverse data types from sensors, social media, logs, and traditional databases through one unified system rather than multiple specialized warehouses.

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Adaptability or versatility

If Data Lakes are used to explore data in unconventional ways, then data exploration flexibility is improved, but security concerns arise and sensitive data may be compromised

Engineering Contradiction:

Improvedata exploration flexibilityVSAvoiddata security

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent applies different security and governance characteristics to different regions of the data architecture. The Data Lake implements role-based access control, data classification, and security policies that are applied locally to specific data zones and user roles. Sensitive data areas have stricter security measures while less sensitive exploration areas have more flexible access, allowing data exploration flexibility where appropriate while maintaining security where required.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent introduces security intermediaries and governance layers between the flexible Data Lake and external access points. These intermediary components include security validation layers, data masking mechanisms, and access control intermediaries that enable flexible data exploration while preventing unauthorized access to sensitive information. The intermediaries act as buffers that maintain security protocols even as data exploration flexibility increases.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS11500826B2System for translating data, extracted from disparate datasources, based on a defined metadata and schema

Publication Date: 2022.11.15 HCL TECH LTD

AI summary

Disclosed is a system for translating data, extracted from disparate data sources, into a homogeneous dataset to provide meaningful information. The database schema definition module defines a database schema in order to extract meaningful information pertaining to a specific use-case. The data source determination module determines one or more disparate data sources pertinent to extract the meaningful information. The data extraction module extracts heterogeneous dataset from the one or more disparate data sources. The data extraction module further passes the heterogeneous dataset to a Data-Translate Markup Language (DTML) executer to translate the heterogeneous dataset into a homogeneous dataset. The data translation module translates the heterogeneous dataset into the homogeneous dataset by using at least one data adapter. In one aspect, the heterogeneous dataset may be translated to perform data analytics on the homogeneous dataset in order to provide the meaningful information pertaining to the specific use-case.