Domain Data Management Services for Consistent Cross-Domain Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In data management systems, particularly in the oil and gas field, transformed data often cannot be reverted to its source form, leading to issues with data consistency and redundancy across different applications and domains, as changes in one domain may not be reliably reflected in others.
Innovation Solution
A method is introduced that registers data files by storing metadata, including a file storage service and a parser, to generate content from raw data, and uses a search service to update related data files across domains based on the content and lineage of the original data, ensuring data consistency and accessibility.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is transformed by applications to generate insights, then data processing capability is improved, but the transformed data cannot be reverted to source data, causing data consistency issues across domains
Solution Approach 1:
The system segments data into two distinct parts: raw data stored in a data store and transformed content stored separately in a content store. This segmentation allows the raw data to serve as the source of truth while transformed content can be generated and updated independently, resolving the consistency issue by enabling selective updates without affecting the original data.
Solution Approach 2:
The system introduces a content store as an intermediary between the data store and applications. The content store holds transformed data representations that can be updated without modifying the raw data in the data store. This intermediary layer enables data transformations to be reversible and consistent across multiple domains and applications.
2Adaptability or versatility
If data is transformed for specific applications, then data utility is improved, but redundant work is conducted when multiple applications need the same data
Solution Approach 1:
The content store serves multiple applications and domains universally. Once data is transformed and stored in the content store, it can be accessed and utilized by any number of applications without requiring separate transformation processes. This multi-functional approach eliminates redundant work while maintaining high data utility across different domains.
3Loss of information
If data transformations are performed to extract insights, then information extraction is improved, but insights cannot be reliably carried through across different applications or domains
Solution Approach 1:
The system implements feedback mechanisms where transformed content in the content store can be updated and these updates are reflected across all applications that access the content store. This feedback loop ensures that insights extracted from data can be reliably carried through and updated across different applications and domains, maintaining information consistency.
Data Source
AI summary
A method includes registering a type of data file. Registering the type of data file includes storing metadata describing the type of data file, the metadata including a file storage service and a parser for the type of data file. The method includes receiving a first data file of the type from a first domain, the first data file having raw data, storing the first data file, storing one or more access rules and a lineage of the first data file, parsing the first data file using the parser to generate a content from the raw data, storing the content separately from the raw data, providing the first data file and the content to a search service, and automatically updating one or more second data files from one or more other domains based on the content of the first data file using the search service and the lineage.