Language Model Training with Non-Linguistic Element Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Language models trained with self-supervised learning on cybersecurity documents are negatively affected by the high presence of non-linguistic elements, which are complex, arbitrary, and meaningless, leading to inefficiencies in learning and performance.
Innovation Solution
A system and method for training a language model that identifies and classifies non-linguistic elements in cybersecurity documents, such as Bitcoin addresses, hash values, and IP addresses, allowing the model to focus on linguistic elements by replacing or masking only completely non-linguistic elements with identification codes, and training to predict and classify these elements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If self-supervised learning is applied to cybersecurity documents containing non-linguistic elements, then the model can process complete documents, but the learning efficiency and model performance deteriorate due to the presence of complex, arbitrary, and meaningless non-linguistic elements
Solution Approach 1:
The patent segments cybersecurity documents into linguistic elements and non-linguistic elements, processing them differently. Linguistic elements are used for self-supervised learning while non-linguistic elements are identified and handled separately through classification tasks, preventing them from degrading model performance
Solution Approach 2:
The patent extracts non-linguistic elements from the document and processes them separately through a classification head, removing their harmful effect on the main language model training while still utilizing them for auxiliary classification tasks
2Reliability
If non-linguistic elements are completely removed from training data, then model performance improves, but the model loses the ability to understand and classify important cybersecurity elements like IP addresses, hash values, and Bitcoin addresses
Solution Approach 1:
The patent creates a multi-functional system where the language model serves dual purposes: maintaining high performance for linguistic understanding while simultaneously acquiring classification capabilities for non-linguistic elements through an integrated classification head
Solution Approach 2:
The patent introduces a classification head as an intermediary component that specifically processes non-linguistic elements, allowing the main language model to focus on linguistic tasks while the classification head handles cybersecurity element identification
Data Source
AI summary
Provided is a system for training a language model for cybersecurity, which includes: a document collection unit that collects a cybersecurity document used for training a language model for cybersecurity; an extraction unit that identifies non-linguistic elements in the cybersecurity document based on a non-linguistic element database; a tokenization unit that tokenizes the cybersecurity document to generate a plurality of tokens; and a language model application unit that controls the language model to simultaneously perform a first task of classifying types of the non-linguistic elements including at least one of a Bitcoin address, a hash value, an IP address, and a vulnerability identifier included in the cybersecurity document and a second task of recovering only linguistic elements of the cybersecurity document.