Separating NOSQL Data and Schema Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Financial and insurance institutions face inefficiencies in managing and analyzing large volumes of complex transaction data across various electronic formats, making it difficult to detect fraudulent or criminal activities due to the lack of effective data storage and management systems.
Innovation Solution
A system and method for capturing and processing data and metadata using a Not Only Structured Query Language (NOSQL) database, where data is stored in a separate NOSQL database and schema is stored in a metadata store, allowing for efficient analytical processing and reducing the need for maintaining multiple data sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is stored in traditional relational databases with multiple data sources, then data can be organized in structured formats, but it becomes difficult to analyze large volumes of complex transaction data across various electronic formats
Solution Approach 1:
The patent implements a universal data lake architecture that can store and process multiple types of data formats (structured, semi-structured, and unstructured) in a single repository. The system uses a common schema framework that adapts to different data sources including transaction logs, flat files, spreadsheets, and databases, eliminating the need for separate processing systems for each format and enabling unified analysis across all data types
Solution Approach 2:
The system dynamically adjusts data organization parameters based on the source and type of data being ingested. It automatically detects data formats and applies appropriate schema mappings, transforming diverse electronic formats into a standardized internal representation that maintains the original data characteristics while enabling consistent query and analysis operations across all data types
2Quantity of substance
If data is stored in a single NOSQL database, then storage footprint is reduced and both applications and analytical processing can run from one system, but schema management becomes more complex
Solution Approach 1:
The patent extracts schema definitions from the data storage structure and manages them separately through dedicated schema registration and validation components. This separation allows the NOSQL database to store flexible, unstructured data while schema metadata is captured, validated, and maintained independently, reducing the complexity of managing schemas within the database itself while enabling automated schema evolution and version control
3Loss of information
If multiple data sources are maintained for different data formats, then data can be stored in its original format, but the system requires maintaining multiple data sources increasing complexity
Solution Approach 1:
The system creates and maintains copies of data in multiple formats within the same data lake infrastructure. Original data files are preserved in their native formats while parallel copies or representations are stored in standardized formats, allowing the system to query and analyze data in the most appropriate format without requiring separate physical data sources for each format
Data Source
AI summary
A method and/or system includes: adding a new source of data to be stored in the data storage system; obtaining the schema for the new source of data to be stored in the data storage system; storing the data to be stored in the data storage system in a Not Only Structured Query Language (NOSQL) database in the data storage system; and storing schema for the data to be stored in the data storage system in a metadata store in the electronic data storage system wherein the metadata store is separate from the NOSQL database.