Dynamic Parser Factory for High Throughput Data Refinement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data processing systems face challenges in scalability and complexity due to evolving data formats and increasing load, leading to inefficient parsing and duplication of technologies, which hinders the ability to handle large volumes of data without system downtime.
Innovation Solution
A data processing system that re-formats incoming data messages with an identifier and data type, selects appropriate message service queues, and uses a parsing processor to generate output data sets, allowing for dynamic allocation of parsing engines and efficient data extraction based on specific needs, thereby maintaining scalability and adaptability without requiring system downtime.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional parsing methods are used to process evolving data formats, then the system can handle current data structures, but the complexity of parsers increases exponentially as data models grow
Solution Approach 1:
The system segments the monolithic parser into multiple specialized parsing engines, each optimized for specific data formats. The parser factory pattern enables dynamic selection and instantiation of appropriate parsing engines based on data type, avoiding the need for a single complex parser that must handle all formats.
Solution Approach 2:
The system changes the parameter of parser selection from static (compile-time) to dynamic (runtime). The parser factory receives data type parameters and dynamically instantiates the appropriate parsing engine, allowing the system to adapt to evolving data formats without increasing overall parser complexity.
2Productivity
If data processing systems fragment business applications to handle increasing load, then scalability improves, but duplication and siloing of technologies occur
Solution Approach 1:
The parser factory serves as a universal component that can instantiate multiple types of parsing engines through a common interface. This multi-functional design allows the system to scale by adding new parsing engine types without creating duplicate specialized processing paths, eliminating technology silos while maintaining scalability.
Solution Approach 2:
The parser factory acts as an intermediary between the business application and multiple parsing engines. It abstracts the complexity of having multiple parsing technologies, providing a unified interface that prevents technology duplication at the application level while enabling scalable deployment of specialized parsers.
3Stability of the object's composition
If compile-time data model definitions are used, then system structure is stable, but the system cannot adapt to evolving data formats without recompilation
Solution Approach 1:
The system transitions from static compile-time data model definitions to dynamic runtime model instantiation. The parser factory can dynamically create and configure parsing engines based on runtime data type information, allowing the system structure to remain stable while adapting to evolving data formats through runtime flexibility.
Solution Approach 2:
The parser factory implements self-service by automatically selecting and instantiating the appropriate parsing engine based on data type parameters without requiring external configuration or recompilation. This autonomous adaptation mechanism maintains system stability while enabling continuous adaptability to new data formats.
4Measurement precision
If sophisticated parsers are implemented to handle complex data models, then parsing accuracy improves, but processing speed decreases due to increased complexity
Solution Approach 1:
Instead of using a single sophisticated parser for all data types, the system applies local quality by creating specialized parsing engines optimized for specific data formats. Each parsing engine has tailored logic for its designated format, achieving high parsing accuracy for that specific type while maintaining fast processing speeds without the overhead of general-purpose complex parsing logic.
Data Source
AI summary
Methods, systems, and articles of manufacture consistent with the present invention provide a data processing system comprising a business application that receives data messages from a plurality of client data sources. The business application comprises a message pre-processor and a parsing processor. The message pre-processor classifies and identifies the data messages and sends the messages in a structured format to a message queue corresponding to its data type. The parsing processor receives the data messages from the message queues and selects a parser by applying a set of parsing rules. The parsing rules apply information about the data message and provide a decision as to the best parsing engine to use out of a plurality of paring engines. The parsing engines are also able to perform information refinement in accordance with selected components defined in a target output data model.