Oilfield Data Pipeline for Domain-Driven Product Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Oilfield entities face challenges in generating and managing data models from disparate data sources, requiring effective communication between domain experts and software experts, and there is a need for a system that allows domain experts to have enhanced control over data application development without requiring software expertise.
Innovation Solution
A data product pipeline system that allows domain experts to develop data products using artificial intelligence and off-the-shelf code units, enabling extraction, transformation, and storage of data without software coding expertise, utilizing a wellsite operations data foundation (WODF) for building and maintaining data pipelines.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data modeling and analysis is accomplished by a combination of domain experts and software applications experts, then data product quality can be improved, but communication complexity and development time increase due to non-overlapping expertise areas
Solution Approach 1:
The patent introduces a data product pipeline as an intermediary system that bridges domain experts and software experts. The pipeline includes extraction modules, transformation modules, and artifact storage that automatically process data according to domain expert specifications without requiring direct software coding expertise from domain experts. This mediator system handles the complex communication and translation between the two expert groups, maintaining data product quality while reducing communication overhead.
Solution Approach 2:
The system enables domain experts to self-serve by directly specifying data extraction and transformation requirements through the data product pipeline interface. Domain experts can define their own data products using domain-specific parameters and criteria without needing to write software code or rely heavily on software experts. The pipeline automatically executes these specifications, allowing domain experts to independently create and manage data products while maintaining high quality through domain-driven specifications.
2Adaptability or versatility
If domain experts require software coding expertise to develop data products, then control over application development is improved, but ease of operation deteriorates due to the learning curve and complexity
Solution Approach 1:
The data product pipeline serves as an intermediary layer that translates domain expert requirements into software operations without requiring domain experts to learn software coding. The pipeline includes pre-built extraction modules, transformation modules, and artifact storage mechanisms that domain experts can configure using domain-specific parameters rather than software code. This maintains full control over application development while preserving ease of operation for domain experts.
Solution Approach 2:
The system replaces the mechanical requirement of software coding with a configuration-based approach. Instead of requiring domain experts to write and maintain software code, the system allows them to configure data products through declarative specifications, parameters, and rules that the pipeline automatically processes. This substitution maintains adaptability and control while dramatically improving ease of operation by eliminating the need for software expertise.
3Ease of operation
If data is stored in raw format from external sources, then data accessibility is improved, but data usability deteriorates due to lack of transformation and processing
Solution Approach 1:
The system performs preliminary data transformation and processing actions as data enters the pipeline from external sources. The extraction module automatically extracts relevant data, and the transformation module applies necessary transformations, filtering, and formatting before storing data in artifact storage. This preliminary action ensures that data is both accessible (stored in organized format) and usable (pre-processed and transformed) simultaneously, eliminating the trade-off between accessibility and usability.
Solution Approach 2:
The data product pipeline establishes a continuous process of data extraction, transformation, and storage. Rather than storing raw data and requiring separate processing steps, the system continuously transforms data into usable formats as it is ingested. This continuous useful action ensures that data remains both accessible and usable at all times, with the pipeline constantly processing and preparing data for consumption without interruption or additional manual intervention.
Data Source
AI summary
A method includes receiving input data from external data sources. The input data is received by a data product pipeline. The method also includes extracting a portion of the input data using the data product pipeline to produce extracted data. The method also includes transferring the extracted data from the data product pipeline to a data product raw storage. The method also includes receiving the input data directly from the external data sources. The method also includes transferring the extracted data and the input data from the data product raw storage back to the data product pipeline. The method also includes receiving data products. The method also includes transforming the input data, the extracted data, and the data products into transformed data using the data product pipeline. The method also includes transferring the transformed data to a data product artifact storage.