Dual-Format Data Storage for MDM Performance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Master Data Management (MDM) systems face challenges in efficiently storing and retrieving dynamic and variable data, such as product information, due to limitations in traditional database schemas, heavyweight XML processing, and proprietary binary formats that hinder performance and data accessibility.
Innovation Solution
A dual-representation approach is implemented, where data is stored in both binary and XML formats, with a binary-to-XML transformer converting binary data into queryable XML for storage, allowing efficient reading, writing, and runtime processing using binary format while maintaining XML retrieval and searching capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If data is stored in vertical table format, then data can be stored in traditional database schema, but many database rows and much server memory are required to reconstruct the tree of data
Solution Approach 1:
The patent creates a copy of the data in a different format (XML representation stored as CLOB) from the original binary format. This copy enables tree structure reconstruction and hierarchical data access without requiring the entire dataset to be loaded into memory, thus reducing server memory requirements while maintaining compatibility with traditional database schemas
Solution Approach 2:
The patent segments the data storage into two distinct representations: binary format for efficient runtime processing and XML format for hierarchical structure and querying. This segmentation allows each format to serve its specific purpose optimally, reducing the need to load all data into memory for tree reconstruction
2Adaptability or versatility
If data is stored in XML format only, then hierarchical document structure is maintained, but loading data from database is too slow and DOM processing uses large amount of memory and CPU resources
Solution Approach 1:
The patent applies different data formats to different operational contexts: binary format is used for runtime processing and data loading where speed is critical, while XML format is used for hierarchical structure representation and querying where adaptability is needed. This local quality assignment resolves the contradiction by optimizing each format for its specific use case
Solution Approach 2:
The system maintains both binary and XML representations of the data. The binary copy enables fast loading and processing, while the XML copy provides hierarchical structure support. This dual-representation approach allows the system to leverage the speed of binary format while maintaining the versatility of XML format
3Speed
If proprietary binary format is used, then quick and easy memory access is achieved, but searching and creating views of data is hindered
Solution Approach 1:
The patent introduces an intermediary mechanism (the application programming interface with dual-format support) that translates between binary and XML representations. This intermediary enables fast binary access for runtime operations while providing XML-based querying capabilities for searching and view creation, thus resolving the contradiction between speed and ease of operation
4Productivity
If dual representation approach is implemented, then performance is enhanced by reducing CPU and memory usage, but data storage complexity increases
Solution Approach 1:
The system creates a copy of data in XML format from the binary format, storing both representations in the database. This copying approach enhances processing performance by allowing the system to use the appropriate format for each operation (binary for speed, XML for querying), while the complexity is managed through automated transformation processes
Solution Approach 2:
The system implements self-service through automated binary-to-XML transformation and automatic format selection by the API. This reduces the operational complexity for users, as the system automatically manages the dual-representation complexity without requiring manual intervention
Data Source
AI summary
Methods and apparatus, including computer program products, implementing and using techniques for managing data items. A data store includes at least two copies of each data item. Each copy of a data item is used for a unique set of operations performed on the data item. An application programming interface receives a request from a consumer to perform an operation on a data item in the data store. The application programming interface selects, based on the received request, which copy of the data item is to be used when performing the operation.