User-Side Scripting for Unstructured Data Indexing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data indexing systems struggle to effectively process and search unstructured and human-generated data, particularly when it is embedded within files or containers, due to limitations in optical character recognition and audio data processing, leading to undetected fraud and unmanaged information.

Innovation Solution

A user-side scripting approach is implemented, allowing users to write and execute scripts that control the indexing engine's operation, enabling the processing of various data types, including unstructured, semi-structured, and structured data, through a user-facing API, which can invoke third-party software for tasks like OCR and audio conversion, and apply heuristics for rapid data analysis without creating indexes.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If traditional indexing engines are used to scan and detect data, then basic file detection capability is provided, but the engine cannot process unstructured data such as flat image text or audio files

Engineering Contradiction:

Improvedata processing capabilityVSAvoidfraud detection accuracy

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent introduces user-side scripts as intermediary components that bridge the gap between the indexing engine and unstructured data formats. These scripts act as mediators that perform specialized processing (OCR for images, audio transcription for audio files) and translate the results into formats the indexing engine can handle, thereby extending processing capability without compromising detection reliability

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system segments the data processing function into two distinct parts: the core indexing engine that handles structured data and user-side scripts that handle unstructured data formats. This segmentation allows each component to specialize in what it does best, with the indexing engine maintaining reliability for detectable data while scripts provide versatility for unstructured formats

Inventive Principle:

Principle #1Segmentation

2Reliability

If comprehensive data scanning is performed to detect all types of fraud, then detection coverage is improved, but processing time and system resources increase significantly

Engineering Contradiction:

Improvefraud detection coverageVSAvoiddata processing time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system applies partial action by selectively invoking user-side scripts only when specific file types or patterns are detected, rather than processing all data uniformly. This approach maintains comprehensive fraud detection coverage by targeting only the necessary data portions with specialized processing, thereby reducing overall processing time while preserving detection reliability

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

User-side scripts perform preliminary processing of unstructured data (converting images to text, transcribing audio to text) before the main indexing engine analyzes the content. This preliminary action prepares the data in advance, allowing the indexing engine to work with standardized formats and improving overall processing efficiency without sacrificing detection coverage

Inventive Principle:

Principle #10Preliminary action

3Adaptability or versatility

If user-side scripting is implemented to extend processing capability, then adaptability to different data formats is improved, but system complexity increases

Engineering Contradiction:

Improvedata format supportVSAvoidsystem architecture complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system implements self-service by allowing users to write and execute their own scripts in familiar programming languages to define custom processing logic. This approach manages complexity by empowering end-users to adapt the system to their specific needs without requiring changes to the core indexing engine architecture, thereby maintaining system simplicity while achieving high adaptability

Inventive Principle:

Principle #25Self-service

4Quantity of substance

If extensive indexing infrastructure is deployed to manage large volumes of data, then data management capability is improved, but infrastructure cost and complexity increase

Engineering Contradiction:

Improvedata processing volumeVSAvoidindexing infrastructure

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The patent substitutes traditional mechanical indexing infrastructure with a more lightweight user-side scripting approach. Instead of relying on complex indexing systems to process all data, the solution uses executable scripts that can be deployed on-demand to process specific data types, thereby managing large volumes of data with reduced infrastructure complexity and cost

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS11727029B2Systems and methods for data indexing with user-side scripting

Publication Date: 2023.08.15 NUIX

AI summary

Systems and methods for electronic data indexing support user-side scripting through an application program interface (API). In some aspects, a user-script generates target data using third party software programs. The user-script sends parameters through the API to an indexing engine. The parameters identify the target data to the indexing engine for processing, analyzing, and performing operations.