Human-in-the-loop Data Analysis for Accuracy and Scalability

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Traditional data analysis methods are limited by human error, inability to select correct hypotheses, and ignoring key variables, leading to missed opportunities and inaccurate insights, especially in large or complex datasets, and existing crowdsourcing approaches suffer from inefficiencies in idea validation and reliance on expert analysts.

Innovation Solution

A combined computer-human approach that leverages structured feedback from untrained humans to enhance automated data analysis, allowing for the detection of patterns and insights through voting, tagging, and proposing hypotheses, while minimizing expert knowledge requirements and automating the processing of feedback to produce statistically valid results.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If traditional automated data analysis is used, then analysis speed and scalability are improved, but accuracy and reliability deteriorate due to human error in hypothesis selection and model fitting

Engineering Contradiction:

Improveanalysis speedVSAvoidanalysis accuracy

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent introduces an interactive system where human analysts serve as intermediaries between automated analysis tools and the data. The system allows analysts to iteratively explore data, form hypotheses, and guide automated analysis, combining the speed of automation with the insight of human expertise to improve both productivity and reliability

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system implements continuous feedback loops where automated analysis results are presented to human analysts, who provide feedback by forming hypotheses and selecting models. This feedback drives subsequent automated analysis iterations, improving accuracy while maintaining scalability through the automated feedback processing

Inventive Principle:

Principle #23Feedback

2Reliability

If expert analysts manually review data to select hypotheses, then analysis accuracy improves, but time consumption and cost increase

Engineering Contradiction:

Improvehypothesis selection accuracyVSAvoidanalysis time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system allows analysts to perform partial manual review by selecting from pre-generated hypothesis candidates rather than reviewing all possible hypotheses. This partial action approach maintains sufficient accuracy while dramatically reducing time consumption compared to complete manual analysis

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The automated system performs preliminary analysis to generate candidate hypotheses and models before presenting them to analysts. This preliminary action filters out obviously incorrect options, allowing analysts to focus their expertise on selecting from a refined set of possibilities, thus reducing overall analysis time

Inventive Principle:

Principle #10Preliminary action

3Difficulty of detecting and measuring

If analysts manually explore data subsets to form hypotheses, then pattern detection capability improves, but scalability deteriorates when dealing with large datasets

Engineering Contradiction:

Improvepattern detection capabilityVSAvoidscalability

Core Design Contradiction:

Difficulty of detecting and measuringVSProductivity

Solution Approach 1:

The system replaces the mechanical process of manual data exploration with automated computational methods. Algorithms automatically explore data subsets, calculate patterns, and generate hypotheses, enabling the system to handle large datasets at scale while maintaining pattern detection capabilities that would be impossible for human analysts to achieve manually

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

4Productivity

If automated analysis models are used, then analysis scalability improves, but measurement precision deteriorates when models do not accurately describe the data

Engineering Contradiction:

Improveanalysis scalabilityVSAvoidmodel fit accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The system dynamically adapts the analysis approach based on data characteristics. Automated model selection algorithms evaluate multiple models and select the one that best fits the specific dataset, allowing the system to maintain high scalability while achieving accurate model fitting by adapting to each dataset's unique properties

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS10796232B2Explaining differences between predicted outcomes and actual outcomes of a process

Publication Date: 2020.10.06 SALESFORCE INC

AI summary

Methods for analyzing and rendering business intelligence data allow for efficient scalability as datasets grow in size. Human intervention is minimized by augmented decision making ability in selecting what aspects of large datasets should be focused on to drive key business outcomes. Variable value combinations that are predominant drivers of key observations are automatically determined from several competing variable value combinations. The identified variable value combinations can then be then used to predict future trends underlying the business intelligence data. In another embodiment, an observed outcome is decomposed into multiple contributing drivers and the impact of each of the contributing drivers can be analyzed and numerically quantified—as a static snapshot or as a time-varying evolution. Similarly, differences in observations between two groups can be decomposed into multiple contributing sub-groups for each of the groups and pairwise differences among sub-groups can be quantified and analyzed.