Information Delivery Platform ELT Data Transformation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data warehousing techniques, such as ETL, face limitations in efficiently transforming and loading data for different end appliances or channels, leading to performance and efficiency issues, especially with large volumes of data.
Innovation Solution
The Information Delivery Platform (IDP) provides a system for processing data by extracting raw data from multiple sources, loading and storing it, and transforming it based on consumption patterns, using an ELT process to enhance efficiency and performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional ETL methods are used to transform and load data for different end appliances or channels, then data can be moved from source systems to target storage system, but the data set needs to be transformed or converted multiple times leading to performance and efficiency issues
Solution Approach 1:
The patent applies preliminary action by performing data transformation after loading raw data into the target storage system rather than transforming before loading. The ELT process loads complete raw datasets first, then transforms only the specific subsets needed for different end appliances or channels, avoiding redundant transformations of entire datasets multiple times.
Solution Approach 2:
The patent extracts and transforms only the specific data subsets needed for particular end appliances or channels after the initial data load, rather than transforming complete datasets multiple times. This selective extraction approach reduces the volume of data being transformed in each subsequent operation.
2Adaptability or versatility
If data is transformed multiple times for different end appliances or channels, then data can be adapted to various consumption patterns, but performance and efficiency deteriorate with large volumes of data
Solution Approach 1:
The patent segments the data processing workflow into distinct phases: initial complete data loading, then separate transformation operations for different end appliances or channels. Each channel receives its own customized transformation process applied only to the relevant data subset, rather than repeatedly transforming the entire dataset.
Solution Approach 2:
The patent implements dynamic data transformation where the transformation process adapts to the specific needs of each end appliance or channel. The system dynamically determines which data subsets to transform and how, based on the specific consumption patterns and requirements of each target system.
3Quantity of substance
If traditional ETL processes are used with large volumes of data, then data can be stored in target storage system, but the repeated transformation and conversion operations create performance bottlenecks
Solution Approach 1:
The patent performs the computationally intensive data loading operation first, getting large volumes of raw data into the target storage system in a single bulk operation. Subsequent transformations are performed on smaller subsets of this already-loaded data, reducing the overall processing burden and improving performance.
Data Source
AI summary
Systems and methods for processing data are provided. The system may include at least a processor and a non-transient data memory storage, the data memory storage containing machine-readable instructions for execution by the processor, the machine-readable instructions configured to, when executed by the processor, provide an information delivery platform configured to: extract raw data from a plurality of source systems; load and store the raw data at a non-transient data store; receive a request to generate data for consumption for a specific purpose; in response to the request, select a set of data from the raw data based on a data map; transform the selected set of data into a curated set of data based on the data map; and transmit the curated set of data to a channel for consumption.