Hierarchical Topic Machine Learning for Dark Data Insight Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies face challenges in efficiently processing and extracting insights from large volumes of complex data, particularly 'dark data' which is often neglected or underutilized, making it difficult to derive actionable insights in a timely manner.
Innovation Solution
A method and system utilizing a hierarchical topic machine learning operation to generate cognitive profiles and insights from user interactions, incorporating cognitive learning and inference systems to process and analyze data, including dark data sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional data processing approaches are used on big data, then data processing can be performed with simple tools, but the processing efficiency is insufficient and cannot handle large volumes of complex data within tolerable time intervals
Solution Approach 1:
The patent segments the data processing task into multiple hierarchical levels (topic modeling, cognitive profile generation, cognitive insight extraction). Each level processes data independently and passes results to the next level, enabling parallel processing and improving overall efficiency without requiring a monolithic complex system.
Solution Approach 2:
The patent introduces a hierarchical dimension to data processing by creating multiple levels of abstraction (from raw data to topics to cognitive profiles to cognitive insights). This dimensional transformation allows complex data to be processed through simpler sequential stages rather than requiring a single complex processing step.
2Quantity of substance
If dark data is collected and stored, then more data sources are available for analysis, but the data becomes difficult to access and utilize at the right time and place
Solution Approach 1:
The patent extracts meaningful patterns and insights from dark data through automated topic modeling and cognitive analysis. By extracting high-level cognitive insights rather than dealing with raw dark data directly, the system makes previously inaccessible data useful and actionable without requiring manual access to the underlying complex datasets.
Solution Approach 2:
The patent introduces cognitive profiles and topic models as intermediary representations between raw dark data and final insights. These intermediaries simplify access to dark data by providing structured, interpretable representations that can be queried and analyzed without directly handling the complexity of the original dark data sources.
3Loss of information
If cognitive learning and inference systems are implemented, then actionable insights can be generated from user interactions, but the system complexity increases
Solution Approach 1:
The patent segments the cognitive analysis process into distinct modules: topic modeling, cognitive profile generation, and cognitive insight extraction. Each module performs a specific function and can be independently optimized or replaced, reducing overall system complexity while maintaining high insight extraction quality.
Solution Approach 2:
The patent creates a universal hierarchical framework that can process multiple types of data sources (user interactions, dark data, structured and unstructured data) through the same cognitive analysis pipeline. This multi-functional approach reduces complexity by avoiding separate specialized systems for different data types.
Data Source
AI summary
A method, system and computer readable medium for generating a cognitive insight comprising: receiving training data, the training data being based upon interactions between a user and a cognitive learning and inference system; performing a hierarchical topic machine learning operation on the training data; generating a cognitive profile based upon the information generated by performing the hierarchical topic machine learning operation; and, generating a cognitive insight based upon the cognitive profile generated using the hierarchical topic machine learning operation.