Federated Feature Extractor Training via Local Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current federated learning methods face challenges in training feature extractors using unlabelled data, as they require sharing raw data with servers, which raises privacy concerns and are inefficient in handling heterogeneous datasets.
Innovation Solution
A self-supervised learning approach where user devices perform local data processing using neural networks, clustering representations to generate local centroids, and updating parameters based on global centroids and feature extractors, minimizing cross-entropy between sample and augmentation assignments, while maintaining privacy through local data processing and K-anonymous clustering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If raw data is shared with servers for training, then training effectiveness is improved, but privacy security deteriorates
Solution Approach 1:
The system segments the training process into local and server components. Local devices perform data processing and feature extraction independently, while only aggregated statistics and model parameters are shared with the server. This segmentation allows training effectiveness to be maintained through distributed computation while preserving privacy by preventing raw data exposure.
Solution Approach 2:
The patent introduces an intermediary representation layer between raw data and server processing. Local devices transform raw data into feature representations and statistical aggregates, which serve as intermediaries for server-based training. This intermediary layer enables effective training while maintaining privacy, as the server receives processed information rather than raw data.
2Productivity
If centralized training is used, then training efficiency is improved, but adaptability to heterogeneous data deteriorates
Solution Approach 1:
The system implements local quality by allowing each device to perform data processing and feature extraction according to its specific characteristics and data distribution. Each device maintains its own local model parameters and processing logic, enabling adaptation to heterogeneous data while contributing to the overall training efficiency through distributed computation.
Solution Approach 2:
The patent introduces dynamic model parameter updates where local devices continuously refine their models based on local data characteristics and receive aggregated feedback from the server. This dynamic adjustment mechanism enables the system to adapt to heterogeneous data distributions while maintaining high training efficiency through coordinated distributed learning.
3Object-affected harmful factors
If local data processing is performed, then privacy is preserved, but computational complexity increases
Solution Approach 1:
The system extracts computationally intensive operations from the server and places them at local devices. Feature extraction, data transformation, and local model training are performed locally, while only lightweight aggregation and parameter updates occur at the server. This extraction reduces server complexity while maintaining privacy through local processing.
Solution Approach 2:
Local devices perform self-service by independently processing their own data and maintaining their own model parameters. Each device autonomously performs feature extraction and local training operations, reducing the computational burden on centralized servers while preserving privacy. The system leverages the computational resources of distributed devices for self-contained processing.
Data Source
AI summary
An apparatus, method and computer program is described comprising: obtaining local data comprising one or more samples at a user device; computing representations of at least some of said samples by passing said one or more samples through a local feature extractor; clustering the computed representations to generate local centroids; providing generated local centroids and parameters of the local feature extractor to a server; receiving global centroids and global feature extractor parameters from said server; updating the parameters of the local feature extractor based on the received global feature extractor parameters; assigning selected samples of one or more samples and one or more augmentations of said selected samples to global clusters; and further updating the updated parameters of the local feature extractor using machine learning principles, thereby generating a trained local feature extractor.