Bounded-Page LIST API Checks for Large-Scale File Existence

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Traditional methods of file existence checking in large-scale data platforms are inefficient and costly due to the high number of API calls required, leading to significant network overhead and prolonged execution times, especially when dealing with billions of files.

Innovation Solution

Implementing a list-based file existence checking method that utilizes a bounded page in the working memory to store a sorted list of metadata files, employing a knapsack fitting algorithm to select a subset of metadata files, and using parallel processing with lexical ordering and a fallback mechanism for comprehensive verification.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If traditional individual API call methods are used to check file existence, then comprehensive verification of each file is achieved, but network overhead and execution time increase significantly

Engineering Contradiction:

Improvefile existence verification accuracyVSAvoidexecution time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent segments the file verification process into two distinct phases: (1) batch verification using LIST API calls to check groups of files together, and (2) individual verification using HEAD API calls only for files that fail the batch check. This segmentation allows the system to leverage the efficiency of batch operations while maintaining the reliability of individual verification when needed, thereby reducing overall execution time without compromising verification accuracy.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies partial action by performing comprehensive individual verification only when necessary (i.e., when batch verification fails). Instead of verifying every file individually from the start, the system first performs a lighter batch verification and only escalates to individual verification for specific cases. This approach significantly reduces the number of expensive individual API calls while maintaining verification reliability.

Inventive Principle:

Principle #16Partial or excessive action

2Reliability

If traditional individual API call methods are used to check file existence, then comprehensive verification of each file is achieved, but network overhead and operational costs increase significantly

Engineering Contradiction:

Improvefile existence verification accuracyVSAvoidnetwork overhead

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent merges multiple file verification operations into a single batch operation by using LIST API calls that can verify multiple files simultaneously. Instead of making separate individual API calls for each file, the system combines verification requests into batches, significantly reducing the total number of network transactions and associated overhead.

Inventive Principle:

Principle #5Merging (Combining)

3Productivity

If batch processing with bounded pages is used to reduce API calls, then execution time and resource requirements are reduced, but complexity of implementation increases

Engineering Contradiction:

Improveverification throughputVSAvoidimplementation complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent segments the large-scale verification task into manageable bounded pages, where each page contains a specific number of files to be processed in a single batch operation. This segmentation makes the implementation more tractable by breaking down the complex task of verifying billions of files into smaller, manageable units that can be processed sequentially, reducing memory requirements and simplifying error handling.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent performs preliminary actions by pre-sorting file paths and pre-organizing them into bounded pages before the actual verification process begins. This preliminary organization simplifies the main verification loop by ensuring that files are already grouped in an optimal order for batch processing, reducing the complexity of dynamic grouping during execution and making the overall implementation more straightforward.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS12450028B1File existence checks using list API

Publication Date: 2025.10.21 SNOWFLAKE INC

AI summary

A data platform that performs a file existence check is provided. The data platform creates a bounded page and selects a set of selected metadata files from a set of metadata files, where each selected metadata file includes a set of data file metadata files. Each member of the set of data file metadata files includes a file name of a respective data file. The data platform stores the set of data file metadata files of each selected metadata file in a first sorted list in the bounded page. The data platform retrieves a second sorted list of file names of a set of data files stored on a data storage system. The data platform determines the existence of each respective data file of each member of the set of data file metadata files on the data storage system by comparing the first sorted list to the second sorted list.