Plugin Data Ingestion System for Heterogeneous Sources
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current big data ingestion technologies face challenges in efficiently ingesting and integrating massive, multi-source heterogeneous data from various business systems into a unified resource pool, particularly due to limitations in data source types and network security constraints that hinder direct communication between different networks.
Innovation Solution
A data ingestion system is deployed with read, transmission, and write plugins for each data source and storage component, allowing for efficient data reading, buffering, and writing across different types of sources and networks, enabling the ingestion of multi-source heterogeneous data and facilitating data transmission across multiple networks by cascading data ingestion systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If direct communication between different networks is used for data ingestion, then data transmission efficiency is improved, but network security constraints are violated
Solution Approach 1:
The patent introduces a data ingestion system as an intermediary component deployed at network boundaries. This system reads data from source systems, buffers it in a transmission buffer queue, and writes it to destination systems, thereby mediating data transmission between different networks while maintaining security constraints.
2Adaptability or versatility
If multiple data source types are supported, then data ingestion versatility is improved, but system complexity increases
Solution Approach 1:
The patent implements a universal data ingestion system that can handle multiple data source types through a standardized architecture. The system uses generic read plugins, transmission buffer queues, and write plugins that can be configured to support different data sources and destinations, allowing one system to perform multiple functions without requiring separate specialized systems for each data type.
3Adaptability or versatility
If data is read, buffered and written through plugin architecture, then data ingestion flexibility is improved, but processing time increases
Solution Approach 1:
The patent implements preliminary buffering of data in a transmission buffer queue between the read and write operations. This buffering mechanism allows the system to read data ahead of time, buffer it temporarily, and then write it to the destination when ready, thereby decoupling the read and write operations and reducing overall processing time while maintaining flexibility through the plugin architecture.
Data Source
AI summary
The present disclosure provides a method and apparatus for ingesting data, a device, a storage medium and a program product, and particularly to the field of computer technology, the field of intelligent traffic, and the like. A specific implementation comprises: deploying a first data ingestion system at a first data source, the first data ingestion system comprising a first read plugin, a first transmission plugin and a first write plugin; reading data from the first data source through the first read plugin; writing the data into a first transmission buffer queue through the first transmission plugin; and acquiring the data from the first transmission buffer queue through the first write plugin and writing the data into a first storage component. The data ingestion system in the present disclosure is divided into a read part, a transmission part and a write part. Each part works by means of a plugin.