Schema-Based Dynamic Parse Engine for Multi-Format Messages
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional parse/build engines face performance issues due to their inability to handle multiple data formats dynamically, with interpreter-based engines being slow and requiring recompilation for new formats, and compiled engines being inflexible and unable to support new message types without code changes.
Innovation Solution
A modular parse/build engine that converts messages into a canonical internal format using a schema registry and individually compiled handlers, allowing for dynamic addition of new formats and handlers without recompiling the entire system, and utilizing a fast indexing system for efficient processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If interpreter-based parse/build engines are used to handle multiple data formats, then adaptability is improved, but processing speed deteriorates
Solution Approach 1:
The system segments the parse/build engine into multiple independently compiled handler components, each specialized for a specific data format. This allows the engine to compile and execute format-specific handlers at high speed while maintaining the ability to handle multiple formats through dynamic handler selection based on message type detection.
2Speed
If compiled parse/build engines are customized for high performance, then processing speed is improved, but adaptability deteriorates
Solution Approach 1:
The system implements dynamic adaptability by allowing new data format handlers to be compiled and registered at runtime without recompiling the core engine. The schema registry dynamically stores handler definitions and the engine selectively loads and executes appropriate handlers based on incoming message formats, enabling both high performance and flexibility.
3Adaptability or versatility
If the entire system is recompiled to add new data formats, then adaptability is improved, but system stability and availability deteriorate
Solution Approach 1:
The system segments the parse/build functionality into independent handler components that can be individually compiled and loaded. This allows new data format support to be added by compiling and registering only the specific handler needed, rather than recompiling the entire system, thus maintaining system availability and stability.
4Adaptability or versatility
If a large grammar dictionary is used to interpret multiple message formats, then adaptability is improved, but memory consumption and complexity increase
Solution Approach 1:
The system extracts the grammar interpretation logic from a single large grammar dictionary and distributes it across multiple specialized handler components. Each handler contains only the grammar rules and parsing logic necessary for its specific data format, reducing overall system complexity and memory consumption while maintaining support for multiple formats.
Data Source
AI summary
A parse/build engine that can handle multi-format financial messages. The engine converts the different format messages into a common format, and the common format message is then processed by the business service application. A parser examines the message and determines an appropriate schema for the particular format of message received. The schema is a data structure in a schema registry that includes a grammar structure for the received format as well as pointers to handlers for converting the different fields of the message into the internal message format using the grammar structure (the “grammar” can include field sequence, field type, length, character encoding, optional and required fields, etc.). The handlers are individually compiled. As formats change, new formats or changes to old formats can be dynamically added to the parse/build engine by loading new schema and handlers.