Middle-Tier Data Ingestion Bypassing SQL Engine

Solution Overview

Problem

Conventional relational database systems are unable to efficiently handle the high-volume data ingestion from IoT devices due to resource-intensive processing overhead, leading to delays and latencies, and NoSQL databases lack the functionality and ACID properties of relational databases.

Innovation Solution

Implementing a middle-tier structure that gathers and organizes data records from IoT devices before optimally-sized groups are ingested into a relational database, bypassing standard SQL engine processing through a fast data transfer protocol that writes directly to the disk.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If conventional relational database systems process high-volume data ingestion from IoT devices, then data is stored in the database, but resource consumption and processing time increase significantly

Engineering Contradiction:

Improvedata ingestion volumeVSAvoidresource consumption

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The patent segments the data ingestion process into two distinct paths: a fast path for high-volume IoT data that bypasses the SQL engine and goes directly to storage, and a standard path for other database operations that goes through the SQL engine. This segmentation allows the system to handle high-volume ingestion efficiently while maintaining data integrity through periodic checksums.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces an intermediary component (the fast path ingestion mechanism) that sits between the IoT data sources and the database storage. This intermediary captures IoT data directly for storage without involving the SQL engine, reducing resource consumption while still ensuring data integrity through checksum validation.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If conventional relational database systems process high-volume data ingestion from IoT devices, then data is stored in the database, but processing delays and latencies occur

Engineering Contradiction:

Improvedata ingestion volumeVSAvoidprocessing time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The patent segments the data ingestion process into two distinct paths: a fast path for high-volume IoT data that bypasses the SQL engine and goes directly to storage, and a standard path for other database operations that goes through the SQL engine. This segmentation allows the system to handle high-volume ingestion efficiently while maintaining data integrity through periodic checksums.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent performs preliminary actions by pre-establishing storage structures and checksum mechanisms before data ingestion begins. The fast path is pre-configured to write directly to storage with predetermined checksum validation points, eliminating the need for time-consuming SQL processing during the ingestion phase.

Inventive Principle:

Principle #10Preliminary action

3Productivity

If NoSQL databases are used to handle high-volume data ingestion, then data storage capacity increases, but ACID properties and querying capabilities are reduced

Engineering Contradiction:

Improvedata ingestion volumeVSAvoidACID properties

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent introduces an intermediary component (the fast path ingestion mechanism) that sits between the IoT data sources and the database storage. This intermediary captures IoT data directly for storage without involving the SQL engine, reducing resource consumption while still ensuring data integrity through checksum validation.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent implements a feedback mechanism through checksum validation. Periodic checksums are computed on the fast-path data and compared against stored checksums to detect and correct errors, providing continuous feedback to ensure data integrity without requiring full ACID transaction processing.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS11394794B2Fast ingestion of records in a database using data locality and queuing

Publication Date: 2022.07.19 ORACLE INT CORP

AI summary

Described is a system, method, and computer program product is provided that implements high-volume data ingestion in a relational database system. A middle-tier structure is provided that sits between the IoT data producers and the back-end database system. Data records are gathered together and organized at the middle tier, and groups of those records are ingested on a group-basis into the database in a manner which bypasses standard SQL engine processing.