Asynchronous Big Data Pipeline Management in Spreadsheets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Client-side spreadsheet applications face inefficiencies in processing large datasets due to synchronous execution of computationally expensive tasks, leading to long wait times and limitations in data processing capacity, and existing solutions require time-consuming, burdensome scripting for managing external Big Data pipelines.
Innovation Solution
A method is introduced to manage external processing pipelines asynchronously from within client-side spreadsheet applications, allowing users to initiate complex data processing flows and automatically generate scripts for external Big Data systems, enabling asynchronous execution and integration of results without the need for individual scripting for each system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronous execution is used for spreadsheet tasks, then calculation results are obtained deterministically, but calculation time and power consumption increase significantly
Solution Approach 1:
The patent divides the spreadsheet calculation task into two segments: a local spreadsheet application handles data preparation and task definition, while an external Big Data framework handles the actual computationally expensive processing. This segmentation allows the heavy lifting to occur asynchronously in the cloud, resolving the contradiction between getting reliable results and minimizing calculation time.
Solution Approach 2:
The patent introduces a cloud-based Big Data framework as an intermediary between the local spreadsheet application and the data processing task. This intermediary handles the computationally intensive operations asynchronously, allowing the spreadsheet application to maintain determinism for result retrieval while significantly reducing the time required for actual calculations.
2Quantity of substance
If external Big Data frameworks are used for processing large datasets, then data processing capacity increases, but script development and pipeline management become complex and time-consuming
Solution Approach 1:
The patent creates a universal interface through the spreadsheet application that can initiate data processing pipelines to various external Big Data frameworks (Hadoop, Spark, SQL, etc.) without requiring users to write individualized scripts for each system. The spreadsheet application serves as a multi-functional gateway that handles both data preparation and pipeline initiation, simplifying the complexity of managing different Big Data systems.
Solution Approach 2:
The patent enables self-service by allowing users to define data processing pipelines directly within the spreadsheet application interface. The system automatically translates these high-level definitions into executable scripts for the underlying Big Data frameworks, eliminating the need for users to manually write and manage complex pipeline scripts while still achieving large-scale data processing capacity.
3Adaptability or versatility
If individualized scripts are written for each external Big Data system, then pipeline control is precise, but the process becomes time-consuming and burdensome
Solution Approach 1:
The patent changes the parameters of pipeline definition from low-level script-level details to high-level conceptual specifications within the spreadsheet application. Users define pipelines using familiar spreadsheet terminology and data structures, and the system automatically translates these parameter changes into system-specific scripts, maintaining control precision while dramatically reducing development time.
Data Source
AI summary
The techniques described herein facilitate asynchronous management of external or Big Data processing pipelines from within client-side (or local) spreadsheet applications, e.g., Microsoft Excel®. In an implementation, a method of operating a spreadsheet application service to facilitate management of big data pipelines from within a client-side spreadsheet application is disclosed. The method includes receiving a processing request initiated by a spreadsheet application. The processing request includes instructions for generating one or more big data pipelines between one or more external distributed computing systems. The method further includes translating the instruction into one or more scripts corresponding to the one or more external distributed computing systems, and executing the one or more scripts to direct the one or more external distributed computing systems to automatically generate the big data pipelines.