Self-Service Data Structure Discovery Through Automated Crawling

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Consumers face a time-consuming, manual process when seeking to understand the data structure of databases, data event contracts, or similar, requiring them to manually inspect code repositories and batch file contracts.

Innovation Solution

A self-service data structure discovery system that includes a data crawler to retrieve data from various sources, store it in a data catalog, receive user queries, and present results through a search dashboard, enabling searches by data structure, datatype, and other criteria across layers.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of time

If manual inspection of code repositories and batch file contracts is used to understand data structure, then detailed data structure information can be obtained, but the process is time-consuming and inefficient

Engineering Contradiction:

Improvetime required for data structure discoveryVSAvoiddata structure discovery efficiency

Core Design Contradiction:

Loss of timeVSProductivity

Solution Approach 1:

The system enables self-service data structure discovery by automatically crawling and indexing data structure information from multiple sources (code repositories, batch file contracts, API documentation) without requiring manual inspection. Users can query the indexed data through a search dashboard, allowing the system to serve itself by automating the discovery process that would otherwise require manual human effort.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system performs preliminary action by proactively crawling and indexing data structure information from various sources before users need to search for it. The data crawler continuously monitors and stores data structure metadata in a searchable format, so when users submit queries through the search dashboard, the information is already prepared and indexed for immediate retrieval.

Inventive Principle:

Principle #10Preliminary action

2Productivity

If automated data crawling and indexing is implemented, then data structure discovery efficiency is improved, but system complexity increases

Engineering Contradiction:

Improvedata structure discovery efficiencyVSAvoidsystem architecture complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system applies segmentation by dividing the data structure discovery process into distinct modular components: a data crawler module for collecting information from multiple sources, an indexing module for processing and storing data structure metadata, and a search dashboard module for user queries. This segmentation allows each component to be developed, maintained, and scaled independently, managing overall system complexity.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system uses an intermediary indexing layer that mediates between the data crawler and the search dashboard. The indexer acts as a mediator that transforms raw data from multiple diverse sources into a standardized, searchable format, simplifying the interaction between the crawling system and the user interface while enabling efficient data structure discovery.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Quantity of substance

If comprehensive data sources are crawled from multiple sources, then data completeness is improved, but data processing requirements increase

Engineering Contradiction:

Improvedata coverage from multiple sourcesVSAvoiddata processing complexity

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The data crawler is designed with universality to handle multiple data sources (code repositories, batch file contracts, API documentation) through a unified crawling and indexing framework. The system uses universal data structure parsing capabilities that can process different formats and sources consistently, reducing the need for separate processing logic for each source type while maintaining comprehensive data coverage.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS12405944B2Systems and methods for self-service data structure discovery

Publication Date: 2025.09.02 JPMORGAN CHASE BANK NA

AI summary

Systems and methods for self-service data structure discovery are disclosed. In one embodiment, a method may include: (1) retrieving, by a data crawler, data from one or more data sources; (2) storing, by the data crawler, the data to a data catalog; (3) receiving, at a search dashboard, a query from a user, the query comprising a search field; (4) submitting, by the search dashboard and using a query tool, the query to a data catalog, wherein the query tool searches the search field by data structure, datatype, field journey, primary key details, and/or multi-value column detail across layers, data lineage controls in the data catalog; (5) receiving, at the search dashboard, a result of the query from the data catalog; and (6) presenting, by the search dashboard, the results.