Code Translation for Resilient Distributed Datasets in Cloud Databases

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing cloud-based database systems face challenges in accessing and utilizing distributed datasets due to incompatibilities between Apache Spark and database systems, leading to increased costs and complexity, as well as difficulties in interoperating data formats between Apache Spark and database tables.

Innovation Solution

The implementation of code translation techniques that convert distributed datasets into equivalent SQL query statements, enabling execution on database tables within the cloud database system, thereby facilitating seamless integration and reducing operational complexity.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If Apache Spark is used for distributed data processing, then data processing capabilities are improved, but system complexity and operational costs increase

Engineering Contradiction:

Improvedata processing capabilitiesVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent introduces a translation layer that acts as an intermediary between Apache Spark code and SQL database systems. This translator converts Spark distributed dataset operations into equivalent SQL queries, enabling Spark's powerful data processing capabilities to be executed through standard SQL database infrastructure, thereby reducing system complexity while maintaining processing power.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The translation system enables a single SQL database system to perform multiple functions: it can execute native SQL queries directly and also execute translated Apache Spark code. This multi-functionality eliminates the need for separate Spark cluster infrastructure, reducing overall system complexity while maintaining data processing capabilities.

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Productivity

If Apache Spark clusters are used for distributed dataset processing, then data analytics capabilities are improved, but operational costs increase

Engineering Contradiction:

Improvedata analytics capabilitiesVSAvoidoperational costs

Core Design Contradiction:

ProductivityVSLoss of energy

Solution Approach 1:

The patent creates a translation copy of Apache Spark code that can be executed in the SQL database environment. Instead of running the original Spark code on expensive distributed clusters, the translated SQL version leverages existing database infrastructure, maintaining analytics capabilities while significantly reducing operational costs.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The translation approach replaces expensive, long-lived Spark cluster infrastructure with cheaper, on-demand SQL database execution. The same data analytics functionality is achieved using cost-effective database resources rather than maintaining dedicated Spark cluster hardware and software.

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

3Adaptability or versatility

If Apache Spark and database systems are used together, then data processing flexibility is improved, but interoperability difficulties increase

Engineering Contradiction:

Improvedata processing flexibilityVSAvoidinteroperability complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The translator serves as an intermediary that resolves interoperability issues between Apache Spark and SQL database systems. It automatically handles the conversion of Spark's distributed dataset operations into database-compatible SQL queries, eliminating manual integration complexity while preserving the flexibility to use both systems.

Inventive Principle:

Principle #24Intermediary (Mediator)

4Productivity

If external Apache Spark clusters are used, then distributed dataset processing is improved, but integration complexity with database systems increases

Engineering Contradiction:

Improvedistributed dataset processingVSAvoidintegration complexity

Core Design Contradiction:

ProductivityVSEase of operation

Solution Approach 1:

The patent merges the functionality of external Apache Spark clusters with the database system by implementing a translator within the database environment. This integration allows distributed dataset processing to occur directly within the database system, eliminating the need for separate external Spark clusters and reducing integration complexity.

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS20220413819A1Code translations of resilient distributed datasets in databases

Publication Date: 2022.12.29 SNOWFLAKE INC

AI summary

The subject technology generates, by executing first code in a first programming language, a resilient distributed dataset (RDD). The subject technology generates, by executing the first code in the first programming language, a second RDD based on the generated RDD. The subject technology generates, by executing the first code in the first programming language, a directed acyclic graph (DAG) including the generated RDD and the generated second RDD. The subject technology serializes the DAG. The subject technology generates a database table based on the serialized DAG. The subject technology converts the first code to second code in a second programming language different than the first programming language, and at least one database query in a second format different than the first programming language. The subject technology sends the converted first code and the at least one database query to a database system for execution.