Atomic Function Workflows for Data Flattening
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional ETL systems require individually generated functions for each new data format, making them inflexible and inefficient for accommodating diverse data sets, especially in providing training data for machine learning models.
Innovation Solution
A data warehousing system utilizing workflows formed from a library of atomic functions, allowing for configurable and reusable data transformation processes that can operate concurrently or sequentially, enabling efficient data flattening and preparation of training data for machine learning models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional ETL systems use individually generated functions for each data format, then data transformation can be performed, but the system becomes inflexible and inefficient when accommodating diverse data sets
Solution Approach 1:
The patent applies universality by creating a unified transformation system that uses a standardized set of atomic functions and workflow templates to handle multiple data formats. Instead of generating individual functions for each format, the system uses a universal workflow engine that can process diverse data types through the same infrastructure, thereby improving adaptability while managing complexity.
Solution Approach 2:
The patent segments complex data transformation processes into atomic functions that can be independently selected and combined in workflows. By breaking down transformation logic into discrete, reusable atomic operations, the system achieves flexibility for diverse data formats without increasing overall system complexity, as each atomic function remains simple and well-defined.
2Manufacturing precision
If traditional ETL systems require code changes for each new data format, then precise data transformation can be achieved, but the development time and maintenance effort increase
Solution Approach 1:
The patent applies preliminary action by pre-defining workflow templates and atomic functions that encapsulate common data transformation patterns. These templates are prepared in advance and can be directly applied to new data formats without requiring custom code development, thereby maintaining transformation precision while significantly reducing development and maintenance time.
Solution Approach 2:
The patent uses copying by allowing workflow templates to be replicated and reused across different data formats. Instead of writing new transformation code for each format, the system copies and adapts existing validated workflows, ensuring consistent transformation precision while eliminating redundant development effort.
3Productivity
If the system uses a library of atomic functions with predefined sequences, then data transformation becomes more efficient and reusable, but the initial system setup and library creation require significant effort
Solution Approach 1:
The atomic function library is designed to be universally applicable across multiple data formats and transformation scenarios. This universality means that while initial library creation requires effort, the same library serves multiple purposes and data types, rapidly amortizing the initial setup cost and improving long-term productivity.
Solution Approach 2:
The system manages the complexity of library creation through parameterization, where atomic functions accept configurable parameters to adapt to different data formats. This approach reduces the number of unique functions needed, as a single atomic function can handle multiple formats by changing its parameters, thereby improving productivity while reducing the burden of system setup.
Data Source
AI summary
Systems and method are provided for data flattening. A corpus of data is extracted from at least one data source and stored at a data warehousing platform. A workflow is applied to the extracted corpus of data to provide a transformed corpus of data. The workflow includes a sequence of atomic functions selected from a library of atomic functions to perform an associated task on the corpus of data. The transformed corpus of data is provided from the data warehousing platform to a machine learning model as a set of training data.