Unified Search for Raw Data and Processing Documentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face challenges in making data processing operation documentation and raw data searchable from a single source, due to different storage requirements and access restrictions, leading to complexity and resource inefficiency.
Innovation Solution
A system that stores raw data in an archival database with fixed storage duration and restricted access, and data processing operation documentation in a separate database with variable storage duration, allowing both to be searchable via a single graphical user interface (GUI) associated with a data searching application.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If raw data is stored in an archival database with restricted access parameters, then data integrity and security are improved, but ease of access and searchability deteriorates
Solution Approach 1:
The system segments data into two distinct databases: an archival database for raw data with restricted access, and a processing database for processed data with broader access. This segmentation allows each database to have optimized access parameters, resolving the contradiction between data integrity and ease of access.
Solution Approach 2:
The system introduces a data processing operation as an intermediary layer between raw data in the archival database and user queries. This intermediary processes and transforms raw data into processed data, enabling users to access information without directly interacting with the restricted archival database.
2Ease of manufacture
If data processing operation documentation is stored separately from raw data, then ease of processing and repeatability are improved, but device complexity and searchability increase
Solution Approach 1:
The system merges the archival database and processing database into a unified search interface through the data searching application. This allows users to search for both raw data and processed data from a single source, reducing the complexity of navigating multiple separate systems while maintaining separate storage for processing purposes.
Solution Approach 2:
The data searching application provides universal access to both databases, serving multiple functions: searching raw data, searching processed data, and retrieving data processing operation documentation. This multi-functionality simplifies the user experience despite the underlying system complexity.
3Productivity
If fixed storage duration is applied to raw data, then data management and security are improved, but loss of information may occur
Solution Approach 1:
The system applies different storage duration policies dynamically to different data types: fixed storage duration for raw data in the archival database to ensure proper data management and security, and variable storage duration for processed data in the processing database to retain information as long as needed for processing and reference.
Data Source
AI summary
In some implementations, a device may store a set of data in a first database associated with a fixed storage duration and a storage parameter that restricts access associated with adding or modifying information stored in the first database. The device may process a subset of data from the set of data, wherein processing the subset of data includes performing one or more data processing operations performed via data processing operation documentation that defines the one or more data processing operations. The device may store the data processing operation documentation in a second database associated with a variable storage duration for data stored in the second database. The device may provide the first database and the second database as data sources for a data searching application to enable the first database and the second database to be searchable via a dashboard associated with the data searching application.