Notebook Snapshot Restore for Serverless Idle Termination

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In serverless computing systems, idle termination of notebooks results in the loss of in-progress work due to automatic shutdown, requiring users to restart from the beginning, leading to resource wastage and inefficiency.

Innovation Solution

A snapshot mechanism is implemented that captures the memory state of a notebook before idle termination, allowing users to resume their work from where they left off by restoring the notebook from the recorded snapshot.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of energy

If idle termination is implemented to conserve cloud system resources, then resource utilization is improved, but user work is lost requiring restart from beginning

Engineering Contradiction:

Improvecloud system resourcesVSAvoidtime to re-execute notebook

Core Design Contradiction:

Loss of energyVSLoss of time

Solution Approach 1:

The system performs preliminary actions by detecting when a notebook is about to become idle and proactively saving the execution state to persistent storage before termination occurs. This allows the notebook to be quickly restored from the saved state rather than re-executing from the beginning, thus resolving the contradiction between resource conservation and time loss.

Inventive Principle:

Principle #10Preliminary action

2Loss of energy

If computing resources are terminated when idle to reduce costs, then operational expenditure is reduced, but in-progress work is flushed or deleted

Engineering Contradiction:

Improvecomputing resourcesVSAvoidin-progress work

Core Design Contradiction:

Loss of energyVSLoss of information

Solution Approach 1:

The system creates a copy of the notebook's execution state (including variables, dataframes, and code) and stores it in persistent storage before terminating the computing resources. When the notebook is accessed again, this copied state is restored, allowing the user to resume work without loss while still achieving resource termination for cost reduction.

Inventive Principle:

Principle #26Copying

3Productivity

If serverless notebooks automatically terminate when idle, then resource allocation efficiency is improved, but user workflow continuity is disrupted

Engineering Contradiction:

Improveresource allocation efficiencyVSAvoidworkflow continuity

Core Design Contradiction:

ProductivityVSEase of operation

Solution Approach 1:

The system performs preliminary saving of the notebook state before idle termination occurs. This preliminary action ensures that when resources are allocated efficiently through termination, the user experience maintains continuity through rapid restoration from the saved state, thus resolving the contradiction between resource efficiency and workflow continuity.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS12530267B1Notebook snapshot restore in a serverless setting

Publication Date: 2026.01.20 DATABRICKS INC

AI summary

A data processing service monitors an activity status of a first set of computing devices that executes code in a notebook. The service may determine that the activity status of the first set of computing devices meets a termination condition. The service may generate, prior to termination of the first set of computing devices, a snapshot recording a current code execution progress for the notebook. To generate the snapshot, the service may determine serialized variables included in the current code execution progress and store the generated snapshot in a data store. The generated snapshot may include the determined serialized variables. The service may terminate the first set of computer devices from executing the code in the notebook.