Flexible Data Preparation with Working and Consumption Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Organizations face challenges in managing large datasets across various storage systems and sources, requiring a system to dynamically receive and manage data while ensuring data processing occurs at the consumption layer for flexibility and tailored data sharing.
Innovation Solution
A computer-implemented system that includes a staging database for receiving data, a working database for processing, and consumption databases for querying, with sub-layers accommodating user-specific preferences and data patterns to facilitate efficient data management and querying.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is stored across various storage systems and sources, then data availability and accessibility are improved, but data management complexity and processing overhead increase
Solution Approach 1:
The patent introduces an enterprise consumption layer as an intermediary between data sources and end users. This layer includes a working database that receives data from multiple storage systems, processes it according to business rules, and delivers tailored datasets to users. The intermediary absorbs the complexity of managing diverse data sources while presenting a simplified, standardized interface to consumers.
Solution Approach 2:
The system segments data into distinct layers: raw data storage layer, processing/working layer, and consumption layer. Each layer has specific functions and data formats. The working database segments data into manageable datasets that can be selectively delivered to different consumption databases based on user needs, reducing the complexity of managing monolithic data structures.
2Adaptability or versatility
If bulk data processing occurs at the consumption layer, then data preparation flexibility is improved, but processing time and computational resources increase
Solution Approach 1:
The system performs preliminary data processing and transformation at the working database layer before data reaches the consumption layer. Common operations include data cleaning, validation, formatting, and initial transformations are completed in advance, so that when data is consumed, it is already prepared and ready for use, minimizing processing time at the point of consumption.
Solution Approach 2:
The consumption layer dynamically processes data based on real-time user requests and business rules. The system can adapt processing depth and scope according to individual user needs, allowing flexible data preparation without requiring all data to be processed uniformly, thus optimizing processing time while maintaining flexibility.
3Productivity
If data is tailored to specific user needs, then data relevance and utility are improved, but data segmentation and system complexity increase
Solution Approach 1:
The system implements local quality by delivering customized datasets to different consumption databases based on user-specific requirements. Each consumption database can have its own data formats, filters, and transformations applied locally, allowing tailored data presentation without requiring the entire system to be complex. The working database maintains a standardized interface while allowing local customization at the consumption level.
Data Source
AI summary
A computer-implemented method for managing large datasets is disclosed. The method may include receiving source data from a data source for staging in a staging database. The method may include storing the source data in the staging database in a source data format. The method may include transmitting the source data from the staging database to a working database. The method may include segmenting the source data into one or more datasets within the working database based on one or more data patterns. The method may include transmitting each dataset to an associated consumption database within a sub-layer. The method may include receiving a request to query the associated consumption database. The method may include executing the query and returning a query result.