Map-Based Energy Data Discovery for Faster Model Development
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge in the energy industry is the inefficient use of data scientists' time due to the high cost of data discovery and preparation, which consumes 80% of their time, leaving only 20% for actual data modeling and machine learning, and identifying suitable and high-quality data for machine learning models is difficult.
Innovation Solution
A method implementing advanced data discovery and visualization for energy data sources, using a map view to select data locations, updating a toolbar with context-sensitive icons, and creating a project workspace for machine learning models, with data automatically imported into the workspace.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional data discovery and preparation methods are used, then data scientists can identify suitable data for machine learning models, but it consumes 80% of their time and reduces productivity
Solution Approach 1:
The system performs preliminary data discovery, evaluation, and preparation before the data scientist begins model development. Automated data agents pre-process data sources, evaluate their suitability, and prepare datasets in advance, so when the data scientist needs data, it is already ready for use. This shifts the timing of data preparation work from during model development to before model development begins.
Solution Approach 2:
The system enables data to serve itself through automated data discovery agents that independently evaluate, filter, and prepare data sources without requiring data scientist intervention. The automated agents perform data quality assessment, source identification, and dataset preparation autonomously, allowing the data infrastructure to self-serve the model development process.
2Reliability
If manual data discovery processes are used, then data scientists can select appropriate data sources, but the process is time-consuming and complex
Solution Approach 1:
The system replaces the manual mechanical process of data discovery with an automated computational system. Data discovery agents use algorithms and automated evaluation metrics to assess data sources, replacing the manual review and selection process. This substitution maintains or improves selection accuracy while dramatically reducing the time required.
Solution Approach 2:
The system introduces automated data discovery agents as intermediaries between data sources and data scientists. These agents act as mediators that evaluate, filter, and recommend suitable data sources based on predefined criteria and data quality metrics, reducing the direct burden on data scientists while ensuring reliable selection.
3Manufacturing precision
If comprehensive data evaluation is performed to ensure high quality data, then suitable data for machine learning can be identified, but the complexity of the process increases
Solution Approach 1:
The system divides the complex data discovery and preparation process into separate modular components: data source discovery agents, data quality evaluation agents, dataset preparation agents, and model training components. Each agent handles a specific aspect of the process independently, making the overall complex system manageable through clear separation of concerns and specialized functionality.
Data Source
AI summary
A method implements advanced data discovery and visualization for energy data sources. The method includes presenting a map view displaying multiple data locations, receiving a selection identifying a subset of the map view, and selecting multiple application components corresponding to the data locations from the subset of the map view. The method further includes updating, responsive to selecting the application components, a toolbar displayed on the map view to include multiple icons corresponding to the application components. The method further includes presenting a visualization component, of the application components, displaying data, corresponding to a data location, in response to a selection from the toolbar. The method further includes creating, using a workflow component of the application components, a project workspace. The data presented with the visualization component is automatically imported to the project workspace.