Process Data Storage for Heterogeneous Instrument Time Series
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional database systems struggle with efficiently storing and querying heterogeneous data sequences from chemical, pharmaceutical, and biopharmaceutical processes, particularly due to inconsistencies in spectroscopic data and the need for predefined schemas, leading to inefficient memory usage and complex queries.
Innovation Solution
A database system that stores heterogeneous data sequences using a schema-on-read approach, allowing flexible storage of data from different scientific instruments without predefined tags, using a NoSQL database with key-value pairs and metadata to facilitate efficient querying across diverse data types.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional time series databases with predefined schemas are used to store process data, then data structure and query capability are maintained, but flexibility in storing heterogeneous data from multiple instruments is reduced and memory usage increases
Solution Approach 1:
The patent implements a dynamic schema approach where the database schema is not fixed in advance but is created automatically based on the actual data structure received from instruments. The system dynamically determines the schema by analyzing the incoming data format, allowing flexible storage of heterogeneous data from multiple instruments without requiring predefined schema definitions.
Solution Approach 2:
The system changes the schema parameter from a static, predefined structure to a dynamic structure that adapts to the actual data. The schema is generated on-demand based on the data received, allowing the database to accommodate varying data formats from different instruments while maintaining query capability through the generated schema.
2Reliability
If spectroscopic data with inconsistent wavelengths is stored using conventional methods, then data can be stored, but data integrity and comparability are compromised
Solution Approach 1:
The patent segments the spectroscopic data by separating the wavelength information from the intensity data. Each data point is stored with its specific wavelength value, allowing the system to handle inconsistent wavelength sampling across different instruments. This segmentation enables later alignment and comparison of spectra despite variations in wavelength positioning.
Solution Approach 2:
The system introduces a metadata layer that acts as an intermediary between the raw spectroscopic data and the stored database records. This metadata includes information about the wavelength range, number of data points, and instrument parameters, allowing the system to track and compensate for sampling inconsistencies while maintaining data integrity.
3Productivity
If multiple separate databases are used to store different data types from various instruments, then each data type can be optimized for storage, but query complexity and system complexity increase
Solution Approach 1:
The patent implements a universal database schema that can store multiple types of data from different instruments using a single database structure. The schema accommodates process data, spectroscopic data, and other instrument data through a unified approach, eliminating the need for separate databases while maintaining efficient storage and querying capabilities.
Solution Approach 2:
The system merges multiple data storage functions into a single database system. By combining process data, spectroscopic data, and instrument data in one unified database with a common query interface, the system reduces overall complexity while maintaining the ability to efficiently retrieve and analyze different data types.
4Measurement precision
If high-resolution spectroscopic data is stored with all data points, then measurement precision is maintained, but storage requirements and memory usage increase significantly
Solution Approach 1:
The patent extracts and stores only the essential information from high-resolution spectroscopic data. Instead of storing every raw data point, the system extracts key features such as wavelength values, intensity values, and metadata about the measurement conditions. This extraction reduces the storage volume while preserving the measurement precision needed for analysis.
Solution Approach 2:
The system applies partial storage by storing only the necessary data points and metadata required for meaningful analysis. The database stores the spectroscopic data in a compressed format that retains essential information while reducing the overall data volume, allowing efficient storage and querying without requiring all raw data to be stored in full resolution.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Aspects relate to a computer-implemented method, a computer program and a system for storing a heterogeneous sequence of discrete-time data determined from a process to produce a chemical, pharmaceutical, biopharmaceutical and/or biological product. The method comprises receiving the discrete-time data, the discrete-time data comprising first data from a first scientific instrument, the first data including a first timestamp corresponding to a first digital signal. The method further comprises receiving second data from a second scientific instrument, the second data including a second timestamp corresponding to a second digital signal. The first scientific instrument differs from the second scientific instrument. The method further comprises storing the first data and first metadata in a first record of a database. The first record comprises a first intensities field having a first data type, and a first descriptors field having a second data type. The method further comprises storing the second data and second metadata in a second record of the database. The second record comprises a second intensities field having the first data type, and a second descriptors field having the second data type. The first metadata includes a first identifier and the second metadata includes a second identifier. When the first data includes first intensities and first descriptors of the first digital signal, storing the first data further comprises storing the first intensities in the first intensities field, and storing the first descriptors in the first descriptors field. When the second data includes second intensities and second descriptors of the second digital signal, storing the second data further comprises storing the second intensities in the second intensities field, and storing the second descriptors in the second descriptors field.