Hybrid Data Normalization for Neural Network Lineage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems, such as distributed data lakes, are computationally expensive, slow to adapt to dynamic environments, lack traceability of data usage, and fail to ensure data integrity and lineage in machine learning applications.
Innovation Solution
A hybrid data normalization process and proof of lineage and integrity system that involves receiving observational data, parsing it into raw data and metadata, storing them separately, and using a DETL request to identify, extract, transform, and load relevant data for neural network training, ensuring data integrity and traceability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If all incoming data is processed immediately via ETL, then data is made available for applications, but computational resources are wasted on data that will never be used
Solution Approach 1:
The patent applies preliminary action by performing data normalization and cataloging at the point of ingestion, before the data is needed for training. This transforms the data into a usable format in advance, so that when training requests occur, the data can be quickly retrieved and applied without performing full ETL processing at that later stage. The normalization happens proactively during ingestion rather than reactively during training.
Solution Approach 2:
The patent segments the data processing workflow into distinct phases: an initial normalization phase during data ingestion that creates a cataloged, normalized version of the data, and a subsequent retrieval phase during training where only the normalized data needs to be accessed. This segmentation allows the computationally intensive normalization to occur once during ingestion, rather than repeatedly during each training operation.
2Ease of manufacture
If static ETL processes are used, then data processing is straightforward, but the system is slow to adapt to rapidly changing end-use applications
Solution Approach 1:
The patent implements dynamics by making the data catalog queryable and adaptable to different training requests. Instead of a static ETL pipeline that processes all data the same way, the system maintains a normalized data catalog that can be dynamically queried based on specific training needs. The system adapts to changing applications by allowing flexible querying and selection from the pre-normalized data, enabling rapid response to new training requirements without reprocessing the underlying data.
3Quantity of substance
If data is stored in a distributed data lake, then large amounts of data can be stored, but there is no traceability of what data is used for what purpose
Solution Approach 1:
The patent introduces an intermediary data catalog that sits between the distributed data lake and the training applications. This catalog maintains normalized versions of the data along with metadata that traces the lineage and usage of each data element. When training occurs, the system queries the catalog to identify which specific data elements are used, creating an audit trail that preserves traceability while allowing the distributed storage architecture to maintain its large storage capacity.
4Duration of action of stationary object
If data is stored in a data lake, then data can be retained for training, but there is no way to know whether the data has been unaltered after storage
Solution Approach 1:
The patent creates a normalized copy of the data during the ingestion phase, storing this copy in a structured catalog format. This normalized copy serves as a reference version that can be used to verify the integrity of the original data stored in the distributed data lake. By maintaining this intermediate normalized representation with associated metadata and checksums, the system can detect whether data has been altered during storage without requiring the original data to remain in its initial format.
Data Source
AI summary
A system, apparatus and method for processing observational data for training a neural network model for use by a neural network. Observational data is parsed into raw data and metadata components and then stored separately. To train the model, a DETL query is used to identify any raw data that may be relevant to training the model. The DETL query is processed by a metadata storage system to match any relevant metadata which, in turn, identifies raw data stored in a raw data storage system. The identified raw data is used to train the neural network model, and a updated neural network model is produced. Each time the neural network model is trained, the relevant raw data and metadata used for each training run is stored in association with the model version so that a lineage of the training may be memorialized.