Pluggable Data Ingestion System with EAV Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data ingestion processes, such as ETL, require predefined extraction logic and storage schema, making them cumbersome and inefficient when data evolves, as they need to be restarted for each change in extracted data or storage format.
Innovation Solution
A system and method for ingesting data in a raw format and formatting it into a common format for storage without prior knowledge of extraction logic or storage schema, using a pluggable architecture with channel adapters and formatters to convert data into key-value pairs for entity-attribute-value storage, allowing flexible data consumption and evolution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If ETL process is used to extract and store data with predefined schema, then data can be accessed in desired format, but the process becomes cumbersome and requires restart when data needs evolve
Solution Approach 1:
The patent segments the data storage system into multiple independent components: raw data storage, extracted data storage, and transformed data storage. This allows each component to operate independently with its own schema, enabling data format changes in one component without affecting others and eliminating the need for process restarts.
Solution Approach 2:
The patent implements dynamic schema evolution by allowing extraction schemas and transformation schemas to be modified independently at runtime. The system can adapt to evolving data needs by updating schemas without restarting the ETL process, making the system flexible and responsive to changing requirements.
2Manufacturing precision
If ETL process is restarted for each change in extracted data or storage schema, then data can be accessed in desired format, but productivity decreases due to frequent restarts
Solution Approach 1:
By segmenting the ETL system into independent extraction and transformation components with separate schemas, the patent enables continuous operation. Each segment can be modified independently without requiring system-wide restarts, thereby maintaining productivity while ensuring data format precision.
Solution Approach 2:
The patent performs preliminary data extraction and storage in raw format with its own schema before transformation. This preliminary action allows schema changes to be made in the transformation layer without affecting the extraction layer, enabling continuous processing and avoiding productivity losses from restarts.
3Stability of the object's composition
If predefined extraction logic and storage schema are used, then data can be systematically organized, but the system becomes complex and rigid when data needs evolve
Solution Approach 1:
The patent divides the data organization system into separate storage components (raw data storage, extracted data storage, transformed data storage) with independent schemas. This segmentation maintains stable organization within each component while reducing overall system complexity by allowing independent modification of each segment's structure.
Solution Approach 2:
The patent creates a universal data storage architecture that can handle multiple data formats and schemas simultaneously across different storage components. This multi-functionality allows the system to maintain stable organization for different data types while adapting to evolving requirements without increasing complexity.
Data Source
AI summary
The present invention extends to methods, systems, and computer program products for managing data ingestion. Aspects of the invention include a pluggable architecture channel service (e.g., a push/pull channel service) to ingest raw data. Aspects of the invention also include a pluggable architecture formatter to convert ingested raw data into a common format, such as, for example, key value pairs. Aspects of the invention also include an EAV storage with functionality allowing consumers to define multiple entities on (and spanning) ingested data sets. Accordingly, data can be ingested without data loss, without having to define extraction logic, and without having to define a storage schema.