Enterprise Consumption Layer for Flexible Data Preparation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Organizations face challenges in managing large datasets across various storage systems and sources, requiring a system to dynamically receive and manage data while ensuring data processing occurs at the consumption layer for flexibility and tailored data sharing.
Innovation Solution
Implementing an enterprise consumption layer that includes a staging database for data organization, a working database for processing, and consumption databases for user-specific queries, with sub-layers accommodating individual user preferences and data patterns, thereby optimizing data access and reducing the need for physical data movement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is stored across various storage systems and sources, then data availability and accessibility are improved, but data management complexity and processing overhead increase
Solution Approach 1:
The patent introduces an intermediary data processing layer between data sources and consumption points. This layer receives data from multiple storage systems, performs unified processing, and distributes it to consumers. The intermediary abstraction layer manages the complexity of multiple data sources while presenting a simplified interface to users, thus resolving the contradiction between data availability and management complexity.
Solution Approach 2:
The system segments data processing into distinct stages: data ingestion, data processing, and data consumption. By dividing the complex data management process into separate functional modules, the patent reduces the complexity burden on individual components while maintaining the ability to handle data from various sources effectively.
2Adaptability or versatility
If bulk data processing occurs at the consumption layer, then data preparation flexibility is improved, but data processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary data processing actions at an intermediate processing layer before data reaches the consumption layer. This preliminary processing includes data cleaning, transformation, and preparation activities that are completed in advance, allowing the consumption layer to focus solely on serving user requests without performing time-consuming bulk processing operations.
Solution Approach 2:
The patent extracts bulk data processing operations from the consumption layer and places them in a dedicated processing layer. This extraction allows the consumption layer to maintain flexibility for tailored data preparation while the processing layer handles time-consuming operations efficiently, separating concerns and optimizing performance.
3Ease of operation
If data is exposed to a large community, then data accessibility is improved, but data security and privacy risks increase
Solution Approach 1:
The patent implements local quality control by providing customized data views and access rights to different user groups. Instead of uniformly exposing all data to everyone, the system tailors data accessibility based on user roles, departments, and security policies. This allows broad data accessibility while maintaining security through localized access controls and data masking mechanisms.
Solution Approach 2:
The intermediary processing layer acts as a security mediator between data sources and users. It enforces access control policies, anonymizes sensitive information, and filters data based on user permissions before data reaches consumers. This intermediary layer enables easy data accessibility while mitigating security risks through centralized security management.
Data Source
AI summary
A computer-implemented method for managing large datasets is disclosed. The method may include receiving source data from a data source for staging in a staging database. The method may include storing the source data in the staging database in a source data format. The method may include transmitting the source data from the staging database to a working database. The method may include segmenting the source data into one or more datasets within the working database based on one or more data patterns. The method may include transmitting each dataset to an associated consumption database within a sub-layer. The method may include receiving a request to query the associated consumption database. The method may include executing the query and returning a query result.