NoSQL Time-Series Storage for Heterogeneous Instrument Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional database systems struggle to efficiently store and query heterogeneous data sequences from scientific instruments in chemical, pharmaceutical, and biopharmaceutical processes, particularly due to inconsistencies and the need for predefined schemas, leading to inefficient memory usage and complex queries.
Innovation Solution
A NoSQL database system with schema-on-read capability stores discrete time data from multiple scientific instruments in a unified format, using key-value pairs and metadata to accommodate diverse data types, allowing flexible storage and querying without predefined schemas.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional database systems with predefined schemas are used to store heterogeneous data from scientific instruments, then data structure and query capability are provided, but memory requirements increase and queries become complex due to schema constraints and data inconsistencies
Solution Approach 1:
The patent implements a dynamic schema approach where the database schema is not fixed in advance but is automatically adapted to the actual data structure. The system uses schema-on-read methodology where the schema is inferred from the data itself during reading operations, allowing flexible storage of heterogeneous data from multiple scientific instruments without predefined structural constraints.
Solution Approach 2:
The patent changes the fundamental parameter of schema definition from static/predefined to dynamic/inferred. By using schema-on-read, the system infers the appropriate schema structure from the actual data being read, allowing the database to adapt its structure based on the data characteristics rather than forcing data into a predetermined structure.
2Ease of operation
If conventional time series databases with predefined schemas are used, then structured query capability is provided, but the ability to handle inconsistent data from different instruments is reduced
Solution Approach 1:
The system dynamically adapts the database schema to match the actual data structure inferred from the data itself. This allows consistent handling of heterogeneous data from different instruments by automatically adjusting the schema to accommodate variations in data format, resolution, and structure without requiring manual schema modification.
Solution Approach 2:
The database system performs self-schema-inference where it automatically determines the appropriate schema structure based on the data it reads. This self-service capability eliminates the need for manual schema definition and allows the system to handle data inconsistencies autonomously by adapting to the actual data characteristics.
3Device complexity
If data is stored with interpolation to fit predefined schemas, then schema compliance is achieved, but data integrity is compromised
Solution Approach 1:
The patent inverts the traditional approach by going from schema-compliance-first to data-integrity-first. Instead of forcing data into a predefined schema and using interpolation to fill gaps, the system reads the data first, infers the appropriate schema from the actual data structure, and then stores the data without any modification or interpolation, preserving complete data integrity.
Solution Approach 2:
The system performs self-schema-inference where the schema is automatically determined from the data itself rather than being imposed externally. This self-service approach ensures that the schema accurately reflects the actual data structure, eliminating the need for interpolation and preserving the original data integrity while achieving schema compliance.
Data Source
AI summary
Aspects relate to a computer-implemented method, a computer program and a system for storing a heterogeneous sequence of discrete-time data determined from a process to produce a chemical, pharmaceutical, biopharmaceutical and/or biological product. The method comprises receiving the discrete-time data, the discrete-time data comprising data from one or more first scientific instruments and including data comprising one or more timestamps corresponding to one or more digital signals.