Semantic Code Search With Globally Mined Hard Negatives
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing contrastive learning frameworks for semantic code search underutilize the bimodal setup of natural language and programming code, leading to inefficient training due to uninformative negative samples and slow convergence, particularly when scaling to large code corpora.
Innovation Solution
A contrastive learning framework that leverages hard negative examples mined globally from the entire training corpus, using K-nearest neighbors of both natural language and programming language parts, and updates the FAISS index periodically to improve representation learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If random negative pairs are used for contrastive learning training, then the training process is simple to implement, but the training efficiency is low and convergence is slow due to uninformative negative samples
Solution Approach 1:
The patent changes the selection criterion for negative samples from random selection to K-nearest neighbor selection based on embedding similarity. This parameter change in the selection strategy transforms uninformative random negatives into informative hard negatives that are semantically similar to the query, thereby improving training efficiency and convergence speed while maintaining implementation feasibility through the use of efficient nearest neighbor search algorithms
Solution Approach 2:
The patent substitutes the mechanical random sampling process with an intelligent selection mechanism based on embedding space proximity. By replacing random negative selection with K-nearest neighbor selection in the embedding space, the system identifies hard negative samples that provide more meaningful gradient signals, improving training efficiency without significantly increasing implementation complexity
2Productivity
If hard negative examples are mined globally from the entire training corpus using K-nearest neighbors, then the training efficiency and convergence speed improve, but the computational complexity and time for mining negative samples increase
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing embeddings for all training samples before the contrastive learning training phase. This preliminary embedding computation allows the system to efficiently perform K-nearest neighbor searches during negative sample mining without repeatedly computing embeddings, thereby reducing the computational complexity during the actual training process while maintaining high training efficiency
Solution Approach 2:
The patent uses copying by creating and storing embedding representations of all training samples in advance. These pre-computed embeddings serve as copies that can be efficiently queried during negative sample mining, avoiding the need to re-process the original code and natural language samples, thus reducing computational complexity while improving training efficiency
3Measurement precision
If the FAISS index is updated periodically, then the representation learning quality improves, but the training time and computational resources increase
Solution Approach 1:
The patent implements periodic action by updating the FAISS index at specific intervals (e.g., every N epochs) rather than continuously or at every training step. This periodic update strategy maintains high representation quality by regularly incorporating new embedding knowledge while avoiding the excessive time cost of continuous updates, thus achieving a balance between representation quality and training time
Solution Approach 2:
The patent applies partial action by updating only the necessary portions of the FAISS index periodically rather than performing full re-computation. This approach maintains high representation quality by updating embeddings with sufficient frequency while reducing the time overhead compared to complete index rebuilds, achieving optimal trade-off between quality and training time
Data Source
AI summary
Embodiments described herein provides a contrastive learning framework that leverages hard negative examples, that are mined globally from the entire training corpus for a given query to improve the quality of code and natural language representations. Specifically, similar examples from the training corpus are extracted and used as hard negatives in an online manner during training while keeping the minibatch construction random.