Decentralized NeRF Training for Private Crowd-Sourced 3D Scenes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural radiance field (NeRF) systems face challenges in processing billions of daily images due to computational intractability and privacy concerns when centralizing personal user images, leading to high server compute and potential exposure of personal content.
Innovation Solution
A decentralized approach, DecentNeRF, where user devices process images locally to separate personal and global 3D data, using secure multi-party computation to aggregate global MLP weights, reducing server computing and ensuring privacy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If centralized NeRF training is used to process billions of daily images, then global 3D scene representation can be achieved, but server computational load becomes intractable
Solution Approach 1:
The system segments the NeRF training process by separating personal MLP weights (kept local on user devices) from global MLP weights (aggregated and stored on server). This segmentation allows the server to only process and store the smaller global representation parameters rather than all individual user images, reducing server computational burden while maintaining global scene representation capability.
Solution Approach 2:
The invention extracts personal content information and keeps it on user devices through local MLP models, removing it from the centralized server processing pipeline. Only the global scene representation is extracted and stored on the server, significantly reducing the volume of data the server must process and store among billions of daily images.
2Quantity of substance
If centralized NeRF training is used to process personal user images, then global 3D scene representation can be achieved, but personal content privacy is compromised
Solution Approach 1:
The system segments the representation learning into personal and global components, with personal MLP weights remaining on user devices and never being transmitted to or processed by the server. This segmentation ensures personal content is never exposed to the server while still enabling global scene representation through aggregated global MLP weights.
Solution Approach 2:
The invention extracts and removes personal content information from the centralized processing pipeline by maintaining local MLP models on user devices. The server only processes and stores global scene representations, effectively taking out personal data exposure from the system architecture.
3Manufacturing precision
If conventional NeRF-W system is used, then high visual quality can be achieved, but server compute resources are excessively consumed
Solution Approach 1:
The system segments computational tasks by performing heavy NeRF training computations locally on user devices for personal MLP weights, while the server only performs lightweight aggregation and storage of global MLP weights. This segmentation maintains high visual quality through comprehensive local processing while dramatically reducing server compute resource consumption.
Data Source
AI summary
In one aspect, a method for rendering of a 3D aggregate image from crowd sourced image data is provided. The method includes receiving, at a server, from each of a plurality of user devices, user global multi-layer perceptron (MLP) weights generated from one or more images of a shared scene. The user global MLP weights are generated so as to not include personal content of a user. The method also includes aggregating the user global MLP weights using secure multi-party computation (SMPC) to further ensure exclusion of personal content. The method also includes sending, from the server to the plurality of user devices, updated weights, wherein the updated weights comprise aggregated global MLP weights. The user devices may then use the updated weights to further help in the implicit separation of personal and global content while retraining of their respective weights on local image data.