Petrotechnical Data Ingestion Framework via REST API Agent Service

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Integrating diverse data sources in data ingestion systems for the oil & gas industry is challenging due to varying interface requirements, data structures, and output formats, often necessitating substantial development efforts and custom code, which leads to delays and potential errors.

Innovation Solution

A data ingestion framework utilizing a REST API agent service with source-specific plugins to interact with multiple data sources, minimizing development efforts by providing common agent functionality and dynamic registration of new data sources.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If custom program code is developed to integrate each new data source, then the system can access and retrieve data from diverse sources with different interface requirements, but the development time and complexity increase substantially

Engineering Contradiction:

Improveability to integrate diverse data sourcesVSAvoiddevelopment complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system segments the data ingestion functionality into a core framework and separate pluggable connectors. Each data source integration is isolated as an independent connector module that implements a standardized interface, allowing new sources to be added without modifying the core system or existing connectors.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The core framework implements universal data ingestion capabilities that work with multiple data sources through a standardized interface. The framework provides common functionalities such as authentication, data retrieval, and error handling that are applicable across all data source types, eliminating the need to rebuild core logic for each new source.

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Adaptability or versatility

If custom program code is developed for each data source, then specific interface requirements can be met, but the time to bring new data sources online is delayed

Engineering Contradiction:

Improvesupport for multiple data source formatsVSAvoidintegration time

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The system performs preliminary configuration by allowing users to define data source connections and credentials through configuration files or UI forms before actual data ingestion begins. Connector modules are pre-built and registered in advance, so when a new data source is added, only configuration is needed rather than full development and testing.

Inventive Principle:

Principle #10Preliminary action

3Adaptability or versatility

If custom program code is written for each data source, then specific data structures and formats can be accessed, but the potential for bugs and errors increases

Engineering Contradiction:

Improvecapability to handle different data formatsVSAvoidsystem stability

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The framework acts as an intermediary layer between the core system and diverse data sources. It provides standardized interfaces and abstraction layers that mediate between different data formats and the uniform internal representation required by the system, reducing errors from direct handling of diverse formats.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The connector modules are designed to be self-configuring and self-validating where possible. They automatically detect data source capabilities, validate configuration parameters, and provide diagnostic information, reducing manual intervention and associated errors.

Inventive Principle:

Principle #25Self-service

4Adaptability or versatility

If substantial development efforts are required to support different data sources, then comprehensive data source coverage can be achieved, but maintenance overhead increases

Engineering Contradiction:

Improvenumber of supported data sourcesVSAvoidmaintenance effort

Core Design Contradiction:

Adaptability or versatilityVSEase of manufacture

Solution Approach 1:

By segmenting each data source integration into an independent connector module, maintenance tasks are isolated to specific modules rather than requiring system-wide changes. This modular structure allows maintenance personnel to work on individual connectors without affecting other parts of the system.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS12093275B2Petrotechnical data ingestion framework

Publication Date: 2024.09.17 SCHLUMBERGER TECH CORP

AI summary

A method, apparatus, and program product utilize a data ingestion framework for petrotechnical data that incorporates a REST API agent service that implements common agent functionality used to interact with various data sources in combination with dedicated source-specific agent plugins to minimize the development efforts required to integrate new data sources into a petrotechnical data ingestion system.