Trajectory Data Tokenization for Language Model Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning technologies lack efficient methods to generate tokens from trajectory data for training language models, especially when dealing with large amounts of data from numerous users.
Innovation Solution
An information processing apparatus and method that encodes continuous position data into character strings, groups these strings, and tokenizes them to efficiently generate tokens from trajectory data for training language models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If trajectory data from a large number of users is collected to train a language model, then the amount of training data increases, but the data processing complexity and noise increase
Solution Approach 1:
The patent segments trajectory data into discrete tokens by dividing continuous position data into discrete position bins and encoding them as character strings. This segmentation transforms complex continuous trajectory data into manageable discrete units that can be efficiently processed by language models, reducing processing complexity while preserving essential movement patterns.
Solution Approach 2:
The patent introduces an intermediary encoding process that converts raw trajectory data into token sequences through multiple transformation steps (position binning, character string encoding, grouping). This intermediary representation layer acts as a mediator between raw data and the language model, simplifying the data structure while maintaining informational content.
2Quantity of substance
If continuous position data is directly used as training data, then the data amount is large, but the processing efficiency decreases
Solution Approach 1:
The patent divides continuous position data into discrete position bins and encodes each bin as a character string token. This segmentation reduces the data processing burden by transforming continuous values into discrete categories, enabling more efficient processing while maintaining the essential spatial information needed for training.
Solution Approach 2:
The patent changes the parameter representation of position data from continuous coordinates to discrete token indices. By transforming the data format through encoding and grouping operations, the system converts high-precision continuous data into a compressed discrete representation that is more efficient for model training while preserving the necessary spatial relationships.
3Adaptability or versatility
If trajectory data is encoded into character strings and tokenized, then the data becomes suitable for language model training, but the processing steps increase
Solution Approach 1:
The patent segments the encoding process into distinct sequential steps: position binning, character string generation, grouping, and tokenization. While this creates multiple processing steps, each step is simple and modular, making the overall process manageable and systematic. The segmentation allows for clear implementation of each transformation stage.
Solution Approach 2:
The patent creates a universal encoding framework that can process any trajectory data into a standardized token format suitable for language models. The encoding unit, grouping unit, and tokenization unit form a multi-functional pipeline that handles various types of position data through a consistent process, enhancing adaptability across different applications.
Data Source
AI summary
An information processing apparatus generates a plurality of character strings by encoding each of a plurality of pieces of continuous position data into a character string, the plurality of character strings each being a character string assigned to a region including a position specified by the respective piece of position data, groups the plurality of character strings into a plurality of groups of character strings, and divides each of the plurality of groups of character strings into a plurality of tokens.