Data-Aware Tokenization for Heterogeneous Computing Data in LLMs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models for analyzing computing data, such as network traffic data, are often limited to specific tasks and require significant human effort and resources to build and train for new tasks, especially when dealing with heterogeneous data, where general text tokenization reduces the context size and loses semantic meaning.
Innovation Solution
The use of data-aware tokenization to generate tokens with semantic meaning for heterogeneous data, enabling the fine-tuning of large language models (LLMs) for in-context learning, which reduces the need for labeled training data and processing resources, and allows for multi-task learning across diverse data types.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If general text tokenization is used for heterogeneous computing data, then the data can be processed by standard NLP models, but the context size is reduced and semantic meaning is lost
Solution Approach 1:
The patent changes the tokenization parameters by introducing data-type awareness, where the tokenizer adapts its behavior based on the detected data type (e.g., IP addresses, URLs, code, natural language). This allows the same tokenizer to preserve semantic meaning for heterogeneous data while remaining compatible with standard NLP model architectures.
Solution Approach 2:
The patent introduces an intermediary data-type-aware tokenizer layer between the raw heterogeneous computing data and the standard NLP model. This intermediary component detects data types and applies appropriate tokenization strategies, preserving semantic information while enabling compatibility with downstream NLP models.
2Measurement precision
If traditional machine learning models are trained for specific tasks on computing data, then task accuracy can be improved, but significant human effort and resources are required to build and train for new tasks
Solution Approach 1:
The patent makes the NLP model universal by enabling it to handle multiple different tasks on heterogeneous computing data through a single fine-tuned model. The data-type-aware tokenization allows the model to automatically adapt to different data types and tasks without requiring separate models for each task.
Solution Approach 2:
The patent performs preliminary action by pre-tokenizing heterogeneous computing data with data-type awareness before fine-tuning the NLP model. This preprocessing step preserves semantic information in advance, allowing the model to learn more effectively from fewer labeled examples and reduce training time for new tasks.
3Measurement precision
If more labeled training data is used to improve model performance on heterogeneous data, then accuracy increases, but the resources and time required for training increase significantly
Solution Approach 1:
The patent changes the tokenization parameters to be data-type aware, which improves the quality of training data representation. This allows the model to achieve better performance with fewer labeled examples because the tokenized input preserves more semantic information and structural characteristics of the heterogeneous data.
Data Source
AI summary
According to examples, an apparatus may include a processor and a non-transitory computer-readable medium on which is stored machine readable instructions that may cause the processor to receive a prompt for a large language model (LLM). The received prompt may include a query to perform a task on computing data through in-context learning in the LLM. The LLM may be fine-tuned on the computing data. In response to the received prompt, the processor may cause the LLM to learn the task via the in-context learning in the LLM. The processor may cause the LLM to output a completion in response to the query for the task. The completion may be generated by performing the learned task on the computing data in the LLM.