Content Source Crawling via Signature Flagging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current file system crawling technologies are inefficient, leading to high costs and risks due to the need to crawl and index vast amounts of unstructured data, which can result in unnecessary storage usage, compliance issues, and security risks, as they are slow and resource-intensive, especially when dealing with massive datasets.
Innovation Solution
A performant content source crawling system that uses file properties and metadata to selectively crawl only modified files since the last index event, employing signature flagging to minimize data processing and reduce computational resources, thereby speeding up the crawl and index update process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If full indexing operation crawls all files to create searchable content index, then search completeness is improved, but time consumption and computing resources increase significantly
Solution Approach 1:
The system performs a preliminary assessment crawl to identify and flag files that need to be re-crawled based on signature changes. This preliminary action filters the file list before the main indexing operation, ensuring only modified files are processed fully, thus reducing overall crawl time while maintaining search completeness for changed content.
Solution Approach 2:
The system extracts only the necessary file metadata (signature, size, modification date) during the assessment phase and separates this from the full content indexing. By taking out only the essential identification attributes, the system can efficiently determine which files require full crawling, reducing the computational burden while preserving complete indexing for relevant files.
2Productivity
If more servers are deployed to increase crawl speed, then productivity is improved, but computing resources and cost increase linearly
Solution Approach 1:
The system dynamically adjusts the crawling strategy based on file signatures and modification dates. Instead of static full-crawl operations, the system adapts its behavior to only process changed files, reducing the computational workload and eliminating the need for linear scaling of servers to maintain productivity.
Solution Approach 2:
The indexing system uses its own assessment of file changes (via signature comparison) to determine what needs processing. This self-service mechanism eliminates the need for external resource scaling, as the system automatically optimizes its own workload based on actual file changes, maintaining high productivity without proportional increases in computing resources.
3Loss of information
If all files are crawled to ensure comprehensive indexing, then information completeness is improved, but unnecessary data processing increases
Solution Approach 1:
The system applies different processing quality to different files based on their signature status. Files with unchanged signatures receive minimal processing (no full crawl), while files with changed signatures receive comprehensive indexing. This local quality differentiation ensures data completeness for modified files while minimizing unnecessary energy consumption on unchanged files.
Solution Approach 2:
The system changes the crawling parameter from 'all files' to 'only flagged files' based on signature comparison results. By adjusting this critical parameter dynamically, the system maintains information completeness for changed content while significantly reducing unnecessary data processing and energy consumption on static files.
Data Source
AI summary
A system and method of improved performant content source crawling. A content source crawling technology that is performant and focused on an initial rapid crawling of a content source to find specific file signatures so that it can determine an inventory of files that have been modified since the last full index event, thereby minimizing the time and computing resources necessary to perform a full crawl on just the select files to update to the search index. A method of Signature Flagging is disclosed and is used to selectively crawl the metadata about contained files and folders found within a content source in order to create that information in an Index within the Shinydocs Search Library.