File Ingestion Error Handling Through Transformation and Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data platforms face challenges in efficiently handling errors during the ingestion of data from external files, particularly in parsing and expression evaluation, with inconsistent error handling across different file formats and a reliance on customized SQL function implementations.
Innovation Solution
Implementing a consistent and generic expression error handling mechanism that decouples error handling from scanner components, using error indicators to manage errors within a Row Set Operator (RSO) framework, allowing for flexible error handling strategies and user-controlled options.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If customized SQL function implementations are used for error handling in different file formats, then error handling can be tailored to specific formats, but device complexity and inconsistency increase across the system
Solution Approach 1:
The patent implements a universal error handling mechanism that works across multiple file formats (CSV, JSON, XML, Parquet) through a single consistent framework. The Row Set Operator framework provides a unified interface for error detection and handling, eliminating the need for separate customized error handling implementations for each file format, thus reducing system complexity while maintaining adaptability.
Solution Approach 2:
The error handling system is segmented into distinct components: scanner components for parsing, projection operators for transformation, and a unified error handling layer. This segmentation allows each component to focus on specific tasks while the error handling framework coordinates them consistently across different file formats, reducing overall system complexity.
2Reliability
If error handling is tightly coupled with scanner components, then error detection can be immediate, but flexibility and maintainability of the system decrease
Solution Approach 1:
The patent extracts error handling functionality from the scanner components and projects it into a separate, unified error handling layer within the Row Set Operator framework. This extraction allows scanner components to focus on parsing while the dedicated error handling layer manages error detection and reporting, improving maintainability while preserving reliable error detection through the projection operator's error indicator arrays.
3Device complexity
If consistent error handling is implemented across all file formats, then system uniformity improves, but the ability to handle format-specific errors may be reduced
Solution Approach 1:
The unified error handling framework provides consistent error detection and handling mechanisms across all file formats through a common interface. The projection operator applies the same error handling logic to all input data regardless of format, ensuring system uniformity while the underlying scanner components maintain format-specific parsing capabilities, thus preserving adaptability to format-specific errors.
Data Source
AI summary
A data platform including an error handling framework for loading of input data. The data platform generates input data columns based on an input file and generates result data columns based on the input data columns and evaluating expressions. The data platform detects projection errors during the generating of the result data columns and stores result error indicators in error indicator arrays of the result data columns based on the projection errors. The data platform generates filtered result data columns based on the result data columns and the result error indicator arrays of the result data columns and stores the filtered result data columns in a database of the data platform.