Transformation Engine for Data Normalization and Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The vast amount of unstructured and incomplete wellness-related data from disparate sources, such as FDA, USDA, and NIH, poses challenges in providing consistent and relevant information for user criteria determination due to proprietary concerns and data inconsistencies.
Innovation Solution
A transformation engine that retrieves data from multiple sources, normalizes it, and applies business rules to filter and format the data, ensuring consistency and relevance for user criteria determination, utilizing interfaces like APIs and ODBC to access various databases including USDA, FDA, and others.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is retrieved from multiple disparate sources, then data completeness is improved, but data consistency deteriorates
Solution Approach 1:
The patent introduces a transformation engine as an intermediary component between disparate data sources and the application layer. This engine receives data from multiple sources (USDA, FDA, NIH, commercial databases) and applies normalization rules, data mapping, and validation to transform heterogeneous data into a consistent format, thereby resolving the consistency issue while maintaining data completeness from multiple sources
Solution Approach 2:
The transformation engine changes the parameters of retrieved data by applying normalization transformations, data type conversions, and format standardization. This allows data from different sources with varying structures and formats to be converted into a unified parameter set, achieving consistency without losing the diversity of source data
2Quantity of substance
If unstructured data is processed, then data coverage is improved, but processing complexity increases
Solution Approach 1:
The transformation engine is segmented into distinct functional modules including data retrieval components, normalization rules engines, data mapping transformations, validation layers, and output generators. This modular segmentation allows each component to handle specific aspects of unstructured data processing independently, reducing overall system complexity while maintaining comprehensive data coverage
Solution Approach 2:
The system performs preliminary actions by pre-defining normalization rules, data mapping templates, and validation criteria before data processing occurs. These pre-configured transformations and rules are stored in the system, allowing unstructured data to be processed through predetermined pathways, thereby reducing processing complexity while maintaining extensive data coverage
3Loss of information
If proprietary data is accessed, then data relevance is improved, but data accessibility deteriorates
Solution Approach 1:
The transformation engine serves multiple functions: it acts as a universal interface that can access both public and proprietary data sources, apply standardized transformations, and output data in a consistent format suitable for various applications. This multi-functionality allows the system to access relevant proprietary data while maintaining ease of operation through a unified access mechanism
Data Source
AI summary
A transformation engine is disclosed that retrieves source data from a plurality of disparate data sources and provides source data that is consistent and normalized. the transformation engine comprises a plurality of interfaces that receive source data from a plurality of disparate databases. The source data comprises a plurality of data elements. The transformation engine further comprises a converter that aggregates and converts the received source data so that the plurality of data elements are recognizable as being substantially equivalent; a data processor that formats the received source data by merging and preparing the received source data; and a business rule applicator that filters the received source data.