Categorical Functions for Parallel Data Processing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing distributed computing solutions, such as MapReduce, are low-level, invasive, and inflexible, requiring specialized expertise and leading to inefficient programming due to round-trip disk I/O and being limited to batch-mode jobs, making them difficult for non-expert programmers and inefficient for real-time processing.

Innovation Solution

The implementation of categorical functions based on category theory to manage all phases of distributed computing, including data division and result combination, providing a platform-agnostic, type-safe, and optimized representation of MapReduce-style programming that hides low-level details and supports both batch-mode and streaming models.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If MapReduce is used for distributed computing, then parallel processing capability is improved, but programming complexity and difficulty increase due to low-level operations and specialized expertise requirements

Engineering Contradiction:

Improveparallel processing capabilityVSAvoidprogramming complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent introduces categorical functions as an intermediary layer between high-level programming intent and low-level MapReduce operations. This mediator abstracts complex distributed computing operations into composable functional primitives (map, reduce, filter, etc.) that follow mathematical laws, shielding programmers from implementation details while maintaining parallel processing efficiency.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent replaces mechanical, imperative programming patterns with functional programming based on category theory. Instead of manually managing state transitions and control flow, the system uses declarative function compositions that automatically parallelize, substituting complex mechanical programming operations with mathematically-grounded functional abstractions.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Productivity

If MapReduce operations are used, then data processing capability is improved, but efficiency deteriorates due to round-trip disk I/O operations

Engineering Contradiction:

Improvedata processing capabilityVSAvoiddisk I/O efficiency

Core Design Contradiction:

ProductivityVSLoss of energy

Solution Approach 1:

The patent applies preliminary action by pre-computing and caching intermediate results in memory during the map phase, allowing subsequent reduce operations to access data without repeated disk I/O. The functional composition model enables optimization where intermediate data structures are maintained in volatile memory across staged computations, eliminating redundant read/write cycles.

Inventive Principle:

Principle #10Preliminary action

3Productivity

If MapReduce framework is used, then distributed computing capability is improved, but flexibility deteriorates due to limitation to batch-mode jobs only

Engineering Contradiction:

Improvedistributed computing capabilityVSAvoidprocessing mode flexibility

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The patent introduces dynamic processing modes through the functional composition framework. The same categorical functions can operate in batch mode (processing complete datasets) or streaming mode (processing continuous data flows), with the execution model adapting to the input data characteristics. This dynamic behavior is achieved through the universal nature of functional composition that works regardless of data arrival patterns.

Inventive Principle:

Principle #15Dynamics

4Productivity

If low-level MapReduce operations are used, then control over distributed computing is improved, but ease of operation deteriorates due to invasive and inflexible programming requirements

Engineering Contradiction:

Improvecontrol over distributed computingVSAvoidprogramming ease

Core Design Contradiction:

ProductivityVSEase of operation

Solution Approach 1:

The patent creates universal categorical functions that serve multiple purposes across different distributed computing scenarios. The same map, reduce, and filter operations work for batch processing, streaming, sorting, aggregation, and transformation tasks. This multi-functionality eliminates the need for specialized low-level operations for each task type, simplifying programming while maintaining control.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS10747571B2Systems and methods of improving parallel functional processing

Publication Date: 2020.08.18 SALESFORCE INC

AI summary

The technology disclosed relates to improving parallel functional processing using abstractions and methods defined based on category theory. In particular, the technology disclosed provides a range of useful categorical functions for processing large data sets in parallel. These categorical functions manage all phases of distributed computing, including dividing a data set into subsets of approximately equal size and combining the results of the subset calculations into a final result, while hiding many of the low-level programming details. These categorical functions are extraordinarily well-ordered and have a sophisticated type system and type inference, which allows for generating maps and reducing them in an elegant and succinct way using concise and expressive programs that can significantly efficientize a whole software development process.