Metadata-Driven Dynamic Reader and Writer Engine
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing systems face inefficiencies when handling large data inputs from various sources with unknown or dynamic schemas, requiring manual and redundant efforts to create reader and writer engines, and lack a general module to handle diverse data formats, leading to processing delays and redundancy.
Innovation Solution
A system that generates dynamic reader and writer engines at runtime using metadata files, enabling the processing of data from various sources with unknown schemas, supporting complex data structures and formats like Cobol copybooks, and optimizing distributed processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If manual processes are used to generate new readers or writers for each schema, then custom data processing can be achieved, but development time and redundancy increase
Solution Approach 1:
The system enables self-service by automatically generating reader and writer engines that can process various data schemas without requiring manual development for each new schema. The generated engines adapt to different data formats autonomously, eliminating the need for developers to manually create custom processing code for each schema variation.
Solution Approach 2:
The patent implements a universal data processing framework that can handle multiple data schemas through a single general-purpose reader and writer engine. These engines are designed to work with various data formats including scalar, map, and list structures, making them multi-functional and applicable across different schema types without requiring separate custom implementations.
2Reliability
If separate custom readers and writers are created for each data format, then data processing accuracy is maintained, but system complexity and redundancy increase
Solution Approach 1:
The system employs universal reader and writer engines that can process multiple data formats including scalar, map, and list structures through a single implementation. These engines maintain data processing accuracy by properly handling different data types and structures without requiring separate custom readers or writers for each format, thereby reducing system complexity while preserving reliability.
Solution Approach 2:
The patent utilizes parameter changes to adapt the reader and writer engines to different data schemas. By dynamically adjusting parameters based on the input data structure, the engines can maintain high processing accuracy across various formats without requiring complex custom implementations for each schema type.
3Ease of operation
If developers hardcode datatypes and structures for their input records, then specialized processing requirements are met, but generalization and reusability are reduced
Solution Approach 1:
The system enables specialized processing capabilities through parameter changes rather than hardcoded structures. Developers can specify processing requirements as parameters that dynamically configure the reader and writer engines, allowing the same engine to adapt to different specialized processing needs without requiring separate custom implementations for each case.
Solution Approach 2:
The patent implements dynamic configuration of data processing engines based on runtime parameters and schema information. This dynamic approach allows the system to meet specialized processing requirements while maintaining generalization and reusability, as the engines can adapt their behavior based on the specific data schema and processing needs rather than being statically configured.
Data Source
AI summary
A method for receiving text files, each having data defined in various format types. The text files are received without a reference to a schema defining the text files. A description file for each of the text files describing one of the format types is received and the description file includes a separate file from a corresponding text file. Before processing the text file and absent receiving the schema, the method identifies a data structure of the description file, the data structure comprising one or more parameters and definitions of the various format types. Each of the text files is processed and during processing of each of the text files, dynamically generating, at run-time, one or more reader engines to read the data in the text files and dynamically extracting, at run-time, data based on the identified data structure. The method generates an output file with the extracted data.