Dual Storage Architecture for Real-Time Manufacturing Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Manufacturing environments, particularly in semiconductor industries, face challenges in managing large volumes of data due to advances in factory automation and the need for real-time monitoring and analysis, which traditional relational database systems are unable to handle efficiently.
Innovation Solution
A method that involves writing data to both a relational database and a distributed storage cluster, with a data management application analyzing queries to determine the optimal storage system for processing based on predefined relationships, allowing for efficient real-time queries and data modeling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional relational database systems are used for data storage and processing, then data consistency and query reliability are maintained, but the system cannot efficiently handle large volumes of manufacturing data
Solution Approach 1:
The patent segments the data storage system into two distinct components: a relational database management system (RDBMS) for structured data and real-time queries, and a distributed file system for large-scale data storage. This segmentation allows each system to operate independently at its optimal capacity, with the RDBMS handling transactional integrity and the distributed system managing bulk data storage, thereby resolving the contradiction between handling large data volumes and maintaining processing efficiency.
Solution Approach 2:
The patent introduces an intermediary layer consisting of ETL (Extract, Transform, Load) processes and data integration mechanisms that bridge the relational database and the distributed file system. This intermediary layer manages data flow between the two systems, ensuring that data is appropriately transformed and loaded according to query requirements, thus enabling efficient handling of large datasets while maintaining the reliability benefits of relational databases.
2Quantity of substance
If distributed file systems are used for processing large data volumes, then extensive data mining capabilities are provided, but real-time monitoring and analysis are inadequate
Solution Approach 1:
The patent applies local quality by assigning different functional characteristics to different parts of the storage system. The relational database portion is optimized for real-time query response with ACID transaction support and indexed access, while the distributed file system portion is optimized for bulk data storage and data mining operations. This localized optimization allows each subsystem to excel at its specific function without compromising the other.
Solution Approach 2:
The patent adds a temporal dimension to the data architecture by maintaining data in both systems with different update frequencies and access patterns. Historical data is archived in the distributed file system for long-term storage and mining, while current operational data remains in the relational database for real-time access. This dimensional separation in time and function enables both real-time monitoring and extensive data mining simultaneously.
3Reliability
If relational databases are used to store large sets of data, then data integrity is maintained, but storage time and query processing time increase significantly
Solution Approach 1:
The patent employs copying by maintaining duplicate copies of data in two different storage systems with different characteristics. The relational database holds a copy optimized for transactional integrity and real-time queries, while the distributed file system holds a copy optimized for bulk storage and analytical processing. This copying strategy allows the system to retrieve data from the appropriate source based on query requirements, reducing overall processing time while maintaining data integrity through the relational database copy.
Data Source
AI summary
Embodiments presented herein provide techniques for managing data in manufacturing systems. One embodiment includes receiving a set of data from a plurality of devices operating in a manufacturing environment. A portion of the set of data is written by a data management application to both a relational database and a distributed storage cluster that includes a plurality of storage nodes in a distributed computing environment. Upon receiving a query to extract a subset of data from the set of data, the query is analyzed to determine attributes of the query. Based, in part on the analysis, one of the relational database and the distributed storage cluster is selected for processing the query.