Custom Data File Generation via Materialized Views
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Extracting custom data sets from large big data environments is resource-intensive and costly due to the significant processing power required, which restricts the frequency and speed of data extraction and transfer, especially when dealing with varied structures and security restrictions.
Innovation Solution
A system that generates and delivers custom data sets by receiving a preselected schema, reading data from multiple sources, filtering columns, and transferring the output data file securely to a client device, utilizing a distributed file system and secure file transfer protocols, while ensuring data readiness and integrity through monitoring and preprocessing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional data extraction methods are used to extract custom data sets from big data environments, then data can be extracted with required security restrictions and permissions, but the processing power required is tremendous which restricts the frequency and speed of extraction and increases processing costs
Solution Approach 1:
The system performs preliminary actions by maintaining materialized views and pre-processed data in the data lake that reflect the current state of source systems. When a data extraction request is received, the system checks whether the required data is already available in the pre-processed materialized views, eliminating the need for comprehensive re-processing of entire big data sets. This preliminary preparation significantly reduces the processing power needed at extraction time while maintaining data freshness and security restrictions.
2Quantity of substance
If comprehensive data sets are extracted to ensure all possible data is available, then data completeness is improved, but the file size and transfer costs increase significantly
Solution Approach 1:
The system extracts only the specific columns and data elements that are actually required by the user's query or business need, rather than extracting entire data sets. The materialized views are structured to contain only relevant data from source systems, and the extraction process selectively retrieves data matching the requested schema. This selective extraction approach ensures data completeness for the specific use case while dramatically reducing file sizes and associated transfer costs.
3Loss of time
If data is frequently extracted to maintain data freshness, then data currency is improved, but the processing costs and resource consumption increase
Solution Approach 1:
The system implements periodic action through scheduled materialized view refreshes that occur at optimized intervals based on data change frequency and business requirements. Rather than continuously processing or extracting data, the system refreshes materialized views periodically when source data changes, maintaining data freshness without the ongoing resource consumption of continuous processing. This periodic approach balances data currency with cost-effective resource utilization.
4Reliability
If security restrictions and permissions are enforced at the column and row level, then data security is improved, but the complexity of data access and extraction processes increases
Solution Approach 1:
The system merges security restrictions and permission checks directly into the materialized view creation and query execution processes. Security policies are embedded in the materialized view definitions, so that when data is pre-processed into materialized views, only authorized columns and rows are included based on user permissions. This merging of security into the core data structure eliminates the need for separate, complex security validation steps during extraction, maintaining high security while reducing process complexity.
Data Source
AI summary
A system for generating and delivering custom data sets in a big data environment may receive a preselected schema that identifies a plurality of columns from a plurality of data sources for inclusion in an output data file. The system reads data from the data sources to generate a data file containing a big data table. The system monitors the plurality of data sources to detect that the data sources have been ingested into a data storage system. The data file is read and a column is filtered from the data file to generate the output data file in response to the preselected schema excluding the column. The output data file is transferred to a client device.