Ingestion Pipeline Document Intelligence Integration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data systems struggle to ingest unstructured file formats, which are not natively supported, leading to limitations in data handling and processing.
Innovation Solution
The use of an external service to process unstructured files, with the database system employing external calls to request the service to process the files and provide results in a specified response schema, including metadata. An external function rowset processor is used to parse the results and generate statistics based on the metadata, allowing the parsed results to be inserted into the target table.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional data ingestion techniques are used, then the system can handle natively supported file formats, but it cannot process unstructured file formats
Solution Approach 1:
The patent introduces an external service as an intermediary component between the database system and unstructured files. This external service acts as a mediator that processes unstructured files and returns results in a standardized schema, allowing the database system to handle diverse file formats without directly implementing format-specific processing logic, thus resolving the contradiction between adaptability and system complexity
Solution Approach 2:
The external service is designed with universal processing capabilities that can handle multiple unstructured file formats through a single interface. By creating a multi-functional processing layer that works with various file types (PDF, images, documents, etc.), the system achieves broad file format compatibility without requiring separate processing paths for each format, maintaining architectural simplicity while enhancing versatility
2Adaptability or versatility
If external services are integrated to process unstructured files, then the system can ingest diverse file formats, but the processing complexity and external dependencies increase
Solution Approach 1:
The external service serves as a specialized intermediary that handles the complexity of unstructured file processing. By offloading format-specific processing to this external component, the core database system maintains its simplicity while gaining the ability to process diverse file formats through standardized external function calls
Solution Approach 2:
The system architecture is segmented into distinct functional components: the database system core, the external service for unstructured processing, and the ingestion pipeline. This segmentation allows each component to specialize in specific tasks, reducing the complexity burden on any single component while enabling comprehensive file format support through coordinated operation
3Quantity of substance
If manual upload techniques are used, then the system can store data in cloud platforms, but it cannot account for different unstructured file formats
Solution Approach 1:
The external service acts as an intermediary processing layer between data upload and storage. It receives unstructured files in various formats, processes them through standardized operations, and converts them into a uniform structure before storage, enabling the system to accept diverse file formats while maintaining consistent data storage capabilities
Solution Approach 2:
The system transforms unstructured files by changing their parameter representation from format-specific structures to a standardized schema. Through parameter transformation and normalization during the external service processing stage, files in different formats are converted into a consistent data structure that can be uniformly stored and queried, resolving the contradiction between storage capacity and format support
Data Source
AI summary
Techniques for ingestion of unstructured files into a target table stored in a network-based database system are described. The unstructured files may include formats not natively supported by a database system. An external service may be used to process the unstructured files. The database system may employ external calls to request the external service to process the unstructured files and provide the results in a specified response schema including metadata.