Data Transformation Engine for Real-Time Derived Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing environments, raw data from various sources is not immediately consumable by client devices due to format incompatibilities, requiring transformation, which leads to latency and stale data issues as derived data is not readily accessible across devices.
Innovation Solution
A computer-implemented method for transforming and distributing data by creating new data types on-demand through derivation and aggregation, using a data transformation engine that generates transformation rules to convert existing data types into requested forms, reducing latency and enabling real-time data consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If data transformation is performed locally on client devices, then data can be consumed in the required format, but latency increases and data becomes stale
Solution Approach 1:
The system performs data transformation in advance by creating derived data types from raw data sources and storing them in a centralized data lake. When a client subscribes to a data type, it is already available for immediate consumption without transformation latency. This preliminary transformation action resolves the contradiction by preparing data beforehand so that format compatibility is achieved without incurring real-time transformation delays.
Solution Approach 2:
The patent introduces a centralized data transformation service and data lake as an intermediary between raw data sources and client devices. This intermediary performs all necessary transformations, filtering, and aggregation centrally, then makes the transformed data available to multiple clients simultaneously. This eliminates the need for each client to perform transformations locally, reducing latency while maintaining format compatibility.
2Ease of operation
If data is transformed and stored locally on each client device, then each client can access its required data format, but data is not readily accessible to other client devices
Solution Approach 1:
The system creates a universal data lake that stores multiple derived data types in a centralized location, making data accessible to any client device that subscribes to the appropriate topics. Instead of each client maintaining its own transformed data locally, the centralized data lake serves all clients universally, enabling both local access efficiency and cross-device sharing simultaneously.
Solution Approach 2:
The patent transitions from a distributed storage model (data scattered across individual client devices) to a centralized storage model (data in a unified data lake). This dimensional change in data architecture allows any client to access any derived data type by subscribing to the appropriate topic, enabling seamless data sharing across devices while maintaining fast access through the centralized structure.
3Adaptability or versatility
If new data types are created by transforming existing data sources, then client devices can consume specialized data formats, but calculation overhead increases
Solution Approach 1:
The system performs data transformation and derivation in advance, creating derived data types from raw data sources before clients need them. These pre-computed derived data types are stored in the data lake and can be directly consumed by clients without requiring real-time calculation. This preliminary action reduces calculation overhead while maintaining data type flexibility.
Solution Approach 2:
The system recycles and reuses intermediate derived data types when creating new derived data types. Instead of always transforming from raw data sources, the system can build new derived types from existing derived types, reducing redundant calculations. This recovering approach minimizes calculation overhead while maintaining the ability to create flexible new data types on demand.
Data Source
AI summary
Various embodiments of the invention disclosed herein provide techniques for transforming and distributing data in a distributed computing system. New data types are created on-demand by deriving, transforming, and aggregating data from already existing data sources. A data transformation engine identifies a first subscription request received from a first subscriber for a first resource included in a plurality of resources. The data transformation engine determines that the first resource is not available from any publisher included in a plurality of publishers. The data transformation engine generates a transformation rule that transforms a set of second resources available from a set of first publishers included in the plurality of publishers into the first resource. The data transformation engine transforms the set of second resources into the first resource based on the inferred transformation rule. The data transformation engine publishes the first resource to the first subscriber.