Chart Regeneration from Metadata for Large Data Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing large data sets from multiple sources becomes challenging due to the difficulty in identifying relevant data and the significant storage requirements for processed data, especially as the volume of raw and analyzed data increases over time.
Innovation Solution
A system and method that allows users to retrieve, process, and analyze large data sets through a user interface, where processed data is generated on demand and not stored automatically, using a user interface with data processing and analysis functions, and storing chart data defining operations and source data identifiers to reduce storage needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If processed data is stored for later access, then data retrieval speed is improved, but data storage requirements increase significantly
Solution Approach 1:
The system creates virtual copies of processed data through metadata descriptions rather than physical storage. The metadata contains sufficient information to regenerate the processed data on-demand, providing fast retrieval without actually storing the large processed datasets.
Solution Approach 2:
The system performs preliminary processing of data characteristics and stores only the essential metadata needed for regeneration. This preliminary action captures the core information needed to reconstruct processed data later, avoiding storage of complete processed datasets.
2Adaptability or versatility
If all raw data is stored for analysis, then data analysis completeness is improved, but system complexity and storage requirements worsen
Solution Approach 1:
The system extracts only the essential metadata from processed data - the minimal information needed to understand and regenerate the data. This extraction separates the critical identification information from the bulk processed data, simplifying storage and access while maintaining analysis capability.
Solution Approach 2:
The metadata acts as an intermediary between the raw data and the processed data requirements. It provides the necessary information to retrieve and regenerate processed data without storing the processed data itself, reducing system complexity while maintaining versatility.
3Productivity
If processed data is generated and stored immediately, then data availability for analysis is improved, but storage costs and resource requirements increase
Solution Approach 1:
Instead of storing actual processed data, the system stores compact metadata copies that describe how to regenerate the processed data. This copying approach maintains data availability for analysis while using minimal storage resources.
Solution Approach 2:
The system changes the storage parameter from storing complete processed datasets to storing only essential metadata parameters. This parameter change reduces storage requirements from gigabytes or terabytes to kilobytes or megabytes while preserving the ability to regenerate processed data when needed.
Data Source
AI summary
A system and method for processing data wherein one or more user selections of source data and an input defining one or more operations to be performed on the selected source data are received to generate processed data for display as a chart; the source data is retrieved from at least one data source, the source data is processed according to the defined one or more operations to generate processed data for output for display as a chart, the chart is stored as data defining the one or more operations and data identifying the source data operated on, a further user selection is received to redisplay the chart; retrieving the source data from the at least one data source; and the source data is processed according to the defined one or more operations to generate the processed data for output for redisplay as the chart.