Automatic Data Ingestion Module for Format-Agnostic Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data extraction and ingestion systems are inefficient in handling and processing data of varying formats, requiring extensive coding and lacking visibility during runtime, and fail to define a common data model across lines of business, leading to increased complexity and time consumption in data analysis.
Innovation Solution
An automatic data ingestion module that utilizes processors and memories to parse and convert data of different formats into a single format, such as JSON, for processing and analysis, implementing a canonical data model and a single data loader to improve processing speed and reduce storage requirements, while using Apache NiFi processes for data ingestion and audit logging.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional data extraction and ingestion systems are used to handle data of varying formats, then data can be processed, but extensive coding is required and processing speed is slow
Solution Approach 1:
The patent implements a universal data ingestion system that can handle multiple data formats (XML, JSON, CSV, mainframe) through a single platform. The system uses format-agnostic parsers and a common data model that works across all formats, eliminating the need for separate coding for each format and enabling single-codebase processing of diverse data sources.
Solution Approach 2:
The patent introduces an intermediary layer consisting of a common data model and format-agnostic parsers that sit between the diverse data sources and the processing layer. This intermediary translates various input formats into a unified internal representation, allowing the rest of the system to process all data types through the same code without format-specific complexity.
2Loss of information
If conventional data extraction systems are used, then data can be loaded, but visibility during runtime and data lineage is not provided
Solution Approach 1:
The patent implements feedback mechanisms through runtime monitoring that tracks data lineage from source to destination. The system provides visibility by logging transformation steps, tracking data flow across processing stages, and maintaining audit trails that show exactly how data moved through the ingestion pipeline, enabling operators to understand data provenance without adding significant complexity.
3Productivity
If multiple specialized data loaders are used for different data formats, then each format can be processed, but time consumption increases and storage requirements grow
Solution Approach 1:
The patent merges multiple specialized data loading functions into a single unified data ingestion platform. By consolidating format-specific loading logic into a common framework with shared parsing and transformation layers, the system reduces redundant processing and eliminates the need for multiple separate loaders, thereby decreasing overall time consumption and storage requirements.
Solution Approach 2:
The patent changes the parameter of data representation by converting all input formats into a unified common data model representation. This parameter transformation allows the system to store and process data in a standardized format, reducing the quantity of data needed for storage and improving processing efficiency by eliminating the need to maintain multiple format-specific representations.
4Adaptability or versatility
If no common data model is defined, then data from different lines of business can be ingested, but defining and processing common entities becomes difficult
Solution Approach 1:
The patent defines a universal common data model that serves as a shared schema across all lines of business. This single data model handles diverse entities from different LOBs through a unified structure, allowing the system to ingest data from multiple sources while maintaining consistency and reducing the complexity of defining and processing common entities across the organization.
Data Source
AI summary
Various methods, apparatuses/systems, and media for implementing an automatic data ingestion module for automatically making data ingestion agnostic to source format are provided. A database stores a plurality of entity data each having a different file format. A processor configures each entity data accessed from the database via a communication network; automatically parses each of the configured entity data; splits and converts the parsed entity data of different file formats into a single file format; and translate the single file formatted entity data into a graph database form for further processing and analysis.