Method and apparatus for rotary position encoding

Abstract

Embodiments of the present application provide a method for rotary position encoding. In a proposed solution of this application, a base value of a frequency is represented in a complex-valued domain. A representation of the base value as complex-valued amplitude improves difference between positions by introducing an additional level of separation (i.e., a phase), and incorporation of both magnitude and phase allows for a richer representation of position information, enabling a model to capture more nuanced relationships and dependencies. This can help mitigate issues related to overlapping representations, thus leading to improved quality metrics.

Description

METHOD AND APPARATUS FOR ROTARY POSITION ENCODINGTECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of a rotary position encoding, and more specifically, to a method and an apparatus for rotary position encoding.BACKGROUND

[0002] Rotary position embedding (RoPE) has emerged as a promising method for encoding position information in sequence-based models, and has shown significant improvements in various natural language processing (NLP) tasks, leading to its widespread adoption. However, as a length of input sequence increases, RoPE begins to exhibit limitations. So, the RoPE mechanism should be adapted to effectively handle extended sequence lengths.

[0003] A position interpolation (PI) method and a neural tangent kernel (NTK) dynamic scaling method are two well-known algorithms proposed to extend a context length beyond a pretrained context window size. However, the PI method still requires finetuning on a large of tokens. Frequency decays faster in NTK dynamic scaling method as the length of a sequence increases, and this can lead to the models being unable to distinguishing information contained in different positions, ultimately impacting its performance on tasks.SUMMARY

[0004] Embodiments of the present application provide a method and an apparatus for rotary position encoding, which allows a richer representation of position information, and leads to improved quality metrics.

[0005] According to a first aspect, there is provided a method for rotary position encoding, including: obtaining a first vector of position information, where the position information is represented based on a first base value of a frequency, and the first base value is represented in a complex domain; and implementing rotary position encoding based on the first vector.

[0006] According to the proposed solution of this application, a base value of a frequency is represented in a complex domain, which allows for more varied representations to be created for adjacent positions and help a model to distinguish the adjacent positions better. This method handles an extended sequence length more effectively without finetuning. Also, this method enables the model to capture more nuanced relationships and dependencies, thus help mitigate issues related to overlapping representations, leading to improved quality metrics.

[0007] In a possible design, where the first base value is represented based on a second base value of the frequency and a scale factor, and the second base value is represented in a real domain.

[0008] In this design, a base value of a frequency in this application is determined based on the base value of the frequency in a traditional approach and a scale factor. This reduces design complexity of the base value of the frequency and implementation complexity when apply this method to varied tasks.

[0009] In a possible design, where the first base value is defined as: b' = b * sa+ l(T, G>-d is an embedding dimension, a is a multiple of reciprocal of b, b' is the first base value, b is the second base value, s is a scale factor, and i is an imaginary unit.

[0010] In a possible design,

[0011] In a possible design, where a rotation matrix RQi<Pimused in the rotary position encoding is represented as:where ch is a hyperbolic cosine function, <pj = Im b'~^ m is an index of a position corresponding to the position information in the first vector, mdj is a rotation angle for position indicated < '

[0012] In a possible design, where a length of the first vector is stretched by a stretching value chcpy.

[0013] In this design, a length of a rotated vector in the RoPE mechanism is stretched by a value greater than one. The value greater than one is obtained as a result of applying mathematical formulas when a base value of a frequency is converted into a complex-valued domain in the present application. This extends traditional approaches, which implies a sum of squares of cosineand sine must be equal to one.

[0014] According to a second aspect, there is provided an apparatus for rotary position encoding. The apparatus has a function of implementing the method in the first aspect and any possible implementation manners of the first aspect. The function includes a corresponding module, unit, or means for performing operations in the first aspect. The module, unit, or means may be specifically implemented by using hardware, may be implemented by using software, or may be implemented by using software in combination with hardware.

[0015] According to a third aspect, another apparatus is described. The apparatus includes a memory and one or more processors. The memory is configured to store a part or all of a necessary computer program or instructions for implementing a function in the first aspect. The one or more processors may execute the computer program or the instructions, and when the computer program or the instructions is / are executed, the communication apparatus is enabled to implement the method in any possible design or implementation of the first aspect.

[0016] In some embodiments, the apparatus may further include an interface circuit, and the processor is configured to communicate with another apparatus or component through the interface circuit.

[0017] In some embodiments, the apparatus may further include the memory.

[0018] The apparatus may be an encoder, a module in an encoder, or a chip responsible for encoding function in an encoder, for example, a modem chip (also referred to as a baseband chip) or an SoC chip or a system in package (SIP) chip that includes a modem module, or an Al module in a communication device such as a terminal, a network device, etc.

[0019] According to a fourth aspect, a computer-readable storage medium is described. The computer-readable storage medium stores computer-readable instructions, and when a computer reads and executes the computer-readable instructions, the computer is enabled to perform the method in any one of the possible designs of the first aspect.

[0020] According to a fifth aspect, this application provides a computer program product. When a computer reads and executes the computer program product, the computer is enabled to perform the method in any one of the possible designs of the first aspect.

[0021] This application encompasses various embodiments, including not only method embodiments, but also other embodiments such as apparatus embodiments and embodimentsrelated to non-transitory computer readable storage media. Embodiments may incorporate, individually or in combinations, the features disclosed herein.DESCRIPTION OF DRAWINGS

[0022] One or more embodiments are exemplarily described by corresponding accompanying drawings, and these exemplary illustrations and accompanying drawings constitute no limitation on the embodiments. Elements with the same reference numerals in the accompanying drawings are illustrated as similar elements, and the drawings are not limited to scale, in which:

[0023] FIG. 1 is a schematic block diagram of RoPE in a transformer architecture.

[0024] FIG. 2 is exemplary flow chart of implementation of rotary position embedding.

[0025] FIG. 3 is an example of applying the RoPE mechanism.

[0026] FIG. 4 is a schematic block diagram of a method (200) for rotary position encoding according to the present application.

[0027] FIG. 5 is an example of an implementation of the proposed amplified RoPE.

[0028] FIG. 6 is an example of a flow chart of the proposed solution used in a transformer architecture.

[0029] FIG. 7 shows visualization of rotation based on the amplified RoPE applied to vector.

[0030] FIG. 8 is an example of an apparatus 10 proposed by the present application.DESCRIPTION OF EMBODIMENTS

[0031] In order to understand features and technical contents of embodiments of the present application in detail, implementations of the embodiments of the present application will be described in detail below with reference to the accompanying drawings, and the attached drawings are only for reference and illustration purposes, and are not intended to limit the embodiments of the present application. In the following technical descriptions, for ease of explanation, numerous details are set forth to provide a thorough understanding of the disclosed embodiments. One or more embodiments, however, may be practiced without these details. In other cases, well-known structures and apparatuses may be shown simplified in order to- simplify the drawings.

[0032] In recent years, rotary position embeddings (RoPE) have emerged as a promisingmethod for encoding positional information in sequence-based models. This method has shown significant improvements in various natural language processing (NLP) tasks, leading to its widespread adoption. However, as a length of input sequences increases, RoPE begins to exhibit limitations. The fixed nature of the rotational encoding' can lead to issues such as difficulty in distinguishing between positions over long sequences, potentially degrading model performance. Consequently, there is a pressing need for alternative approaches that can effectively address these challenges.

[0033] One promising solution is position interpolation (PI), which enhances flexibility of positional encoding by interpolating between known positions. This method allows the model to better adapt to varying sequence lengths, providing a more nuanced representation of positional information. However, it doesn’t take advantages of applying better frequencies and requires additional finetuning to ensure optimal performance. To solve problems of the PI approach, neural tangent kernel (NTK)-scaling method was proposed. Its main idea is the concept of adjusting the base frequency values of positional embeddings according to a specific formula.

[0034] This application is to accelerate the decay of frequencies as the sequence length increases. Specifically, this application generalizes the representation of base value of frequencies to complex domain in order to improve the difference between positions by introducing an additional level of separation.

[0035] Some related prior arts are introduced first to make the solution proposed by this application to be understood easier.

[0036] RoPE is a position-encoding technique, that effectively encodes positional information in transformer-based language models. The RoPE has the following form:> Where 6j is a set of angles and the set is half as large as the embedding size of the tokens, m is a position of a token in the input sequence.

[0037] RQMis a rotation matrix as follow: "

[0038] Where 0= {0j = 100002(^ d is embedding dimension.

[0039] RoPE encodes an absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention formulation.

[0040] FIG.l is a schematic block diagram of RoPE in. a transformer architecture. As shown in FIG. 1, the first step is to obtain a vector of positions, then calculate frequencies, and multiply positional by angles.

[0041] FIG. 2 is an exemplary flow chart of implementation of rotary position embedding. The flow chart may include the following steps: 1) reshaping a source tensor and partitioning the source tensor into pairs; 2) representing each pair as a point on, a complex plane with a reality part as a horizontal axis and an imagination part as a vertical axis; 3) rotating each point on the complex plane by the angle m0j, and obtaining resulting points. 4) representing each resulting point into a pair and forming a final tensor; 5) reshaping the final tensor to a required dimension.

[0042] FIG. 3 is an example of applying the RoPE mechanism. For example, there is a first sentence “a pig chased a dog”. A position of the word “pig” in the first sentence is 2, and the position of the word “dog” in the first sentence is 5. A distance between the two words is 3. In a second sentence “yesterday a pig chased a dog”, we will notice that the position of the word “pig” in the first sentence is 3, and the word “dog” in the second sentence is now 6, however, the distance between the two words remains the same. This is the main idea of the RoPE. Thus, a relative position is maintained: if two words, “pig” and “dog”, maintain the same relative distance in different context, the different vectors are rotated by the same amount. This ensures that the angle, and consequently the dot product between theses vectors, remains constant.

[0043] However, the limitation of current solution is transformer-based language models fails to generalize past the sequence length they were trained on. So, the RoPE mechanism should be adapted to effectively handle extended sequence lengths.

[0044] There are two well-known algorithms that can extend a context window size ofpretrained RoPE based on large language models (LLMs), while demonstrating strong empirical results on various tasks that require long context.

[0045] A position interpolation (PI) method was proposed to extend the context length beyOnogd the pretrained limit. The basic idea is multiplication of position indices to a ratio (which also can be referred to a scale factor) between the extended context length and an original context length. This method normally requires finetuning on about 1 to 10 billion tokens, and the way it modifies the RoPE formula didn’t take advantage of applying better frequencies.

[0046] To solve the problems of the PI method NTK Dynamic Scaling was proposed. The goal of this method is to spread out the interpolation pressure across the hidden dimensions using a base-change instead of scaling the frequencies by a fixed factor s. The lowest frequency needs to be scaled as much as linear positional scaling and the highest frequency to stay constant. A new base b' is, introduced such that the last dimension matches the wavelength of linear interpolation with a scale factor s:db'= b * sa-2 (4)Where s is a scale factor, d is embedding dimension, and b is an original base value.

[0047] This position indices remain unchanged, and the frequencies decrease more quickly.

[0048] However, as the length of a sequence increases, some positions may have identical or very similar representations. This can lead to the model being unable to distinguish between the information contained in different positions, which degrades its ability to generalize. When multiple positional have overlapping representations, the model may struggle to capture the nuanced relationships and dependencies between elements in the sequence, ultimately impacting its performance on tasks that require fine-grained understanding of context.

[0049] In view of this, a method and an apparatus for position encoding are proposed by the present application. In the solution proposed by this application, by using complex representations, both magnitude and phase can be considered. This allows for a richer encoding of positional information, where different angles can represent various positions more distinctly. What’s more, different positions that have identical representations can be avoid.

[0050] This application can be applied in position encoding to avoid poor position discrimination, because the use of the base value of frequencies as the complex amplitude allows for more varied representations to be created for adjacent positions, which help the modeldistinguish adjacent positions better. This provides more effectively handle extended sequence lengths without finetuning.

[0051] The solution of this application will be elaborated in the following.

[0052] FIG. 4 is a schematic block diagram of a method (200) for rotary position encoding according to the present application. The method (200) may be implemented by an apparatus, for example, a terminal or a module in a terminal, a circuit or a chip (for example, a modem chip, also referred to as a baseband chip, or a system on chip (SoC) chip or a system in package (SIP) chip that includes a modem core), etc.

[0053] In addition, this application is applicable to more than one scenario, for example, deep learning models or frameworks for tasks like long text modeling and summarization, tasks involving time series analysis such as forecasting financial data or analyzing sensory information, natural language processing (NLP) tasks such as text generation, sentiment analysis, answers to questions, etc.

[0054] The method (200) includes the following steps 210~220.

[0055] Step 210, obtaining a first vector of position information, where a representation of the position information included in the first vector is based on a first base value of a frequency and the first base value is represented in a complex domain (also referred to as a complex-valued domain, or a complex field).

[0056] In an implementation, the first vector can be a vector of position indexes of tokens in an input sequence.

[0057] In another implementation, an input of the method 200 may be a multidimensional tensor. The tensor can be decomposed into a set of two-dimension vectors, and the method 200 can be applied to each vector from the set.

[0058] Step 220, implementing rotary position encoding based on the first vector.

[0059] The proposed solution may be called as amplified RoPE, and the amplified RoPE involves representation of the first base value as complex-valued amplitude in order to better distinguish positions in the sequence. The incorporation of both magnitude and phase allows for a richer representation of position information, enabling the model to capture more nuanced relationships and dependencies. This can help mitigate issues related to overlapping representation, thus lead to improved quality metrics.

[0060] In a specific implementation, the first base value is represented based on a second base value of the frequency and a scale factor, and the second base value is represented in a real domain. For example, the first base value is represented as follows:b' = b * sffl+ ia(5)

[0061] In formula (5), b' is the first base value, b is the second base value, s is the scale factor, co= d is an embedding dimension, a is a multiple of reciprocal of b. For example, 4a=~b-

[0062] Based on the base value proposed in this application, i.e., the first base value, the rotation matrix is changed as follows:

[0063] In formula (6), m is an index of a position in the first vector, mdj refers to the frequency Oj multiply by the position m, therefore, mdj is a rotation angle and it is a multiple of the position m of the tokens in the input sequence, " ch is a hyperbolic cosine function. The rotation matrix RQi(Pimin formula (6) involves an additional parameter <p compared with therotation matrix RQ,min formula (3) of prior arts.

[0064] FIG. 5 is an example of an implementation of the proposed amplified RoPE. As shown in FIG.5, an encoding process of the proposed solution is as follows: reshaping a source tensor and partitioning the source tensor into pairs. Representing each pair as a point on a complex plane with a reality part as a horizontal axis and an imagination part as a vertical axis. Rotating each point on the complex plane by the angle mdj, then rotating each point in a third dimension by the angel (p, and obtaining resulting points. Then projecting the resulting points back onto the complex-valued domain, representing each resulting point into a pair and forming a final tensor. Reshaping the final tensor to a required dimension.

[0065] The first vector in the proposed solution, for example, in the embodiments of the method 200, is obtained by reshaping the source tensor described in FIG. 5. A vector is a first-order tensor.

[0066] According to the proposed solution (i.e., the amplified RoPE), since the base value b' of frequencies is complex-valued, the frequencies also become complex-valued. The corresponding sines and cosines involved in the rotation matrix also are complex-valued. Theseare the differences of the proposed amplified RoPE of this application and the original RoPE mechanism adopted in prior arts.

[0067] As an example, application of the proposed solution in a transformer architecture is as shown in FIG. 6.

[0068] FIG. 6 is an example of a flow chart of the proposed solution used in a transformer architecture. The proposed position encoding method may be implemented by a processing module integrated in relevant models such as an NLP model, a model designed for solving signal processing tasks, etc. The flow chart of the proposed solution may include the following steps:

[0069] 1) obtain a first vector of position information;

[0070] 2) calculate a new base value b' of frequency that with a complex- valued amplitude according to the formula (5), where the new base value refers to the first base value in embodiments of the method 200;

[0071] 3) generate complex frequencies based on the base value b' of the frequency;

[0072] 4) multiply positions indicated by the position information by obtained complex-valued angles, i.e., mdj

[0073] 5) multiply sines and cosines of a real part, by hyperbolic cosine of an imaginary part, where sines or cosines of the real part also is complex-valued;

[0074] 6) take complex-valued sines / cosines;

[0075] 7) execute the model with an updated rotary position encoding using an attention mechanism, where the attention mechanism can be any kind of attention mechanisms such as selfattention, or flash-attention and etc.

[0076] If the proposed amplified RoPE is used, the distance between the positions of the words “pig” and “dog” in the sentence “a pig chased a dog” in the above example increases. It is because rotated vector lengths are increased as a result of multiplication by the hyperbolic cosine.

[0077] FIG. 7 shows visualization of rotation based on the amplified RoPE applied to vector. It can be found that the rotated vector lengths obtained according to the amplified RoPE is stretched compared with that obtained based on the original RoPE.

[0078] In the proposed application, a value by which to stretch the rotated vectors cannot be an arbitrary number greater than one, because in this case the proposed amplified RoPE will get a deterioration in the quality metric. The value, which is called a stretching value, is obtained on theiobasis of mathematical formulas as follows:< < < < <

[0079] The stretching value refers to ch q)j in the formula (9).

[0080] Transformer-based language models fail to generalize past the sequence length they were trained on. To effectively handle extended sequence lengths, the proposed solution of this application was designed. The main idea is representation of base value of frequencies as complexvalued amplitude. It improves difference between positions by introducing an additional lever of separation (i.e., phase). The proposed solution was tested with various lengths of sequence on WikiText dataset on Falcon-7b, LLAMA-2-7b models, also on 2WikiMQA, TriviaQA, HotpotQA datasets on LLAMA-2-7b-chat model, and compared with well-known approaches, for example, position interpolation, dynamic NTK scaling. The results show that the proposed solution surpasses all the existing methods on both perplexity and Fl metric.

[0081] Some results are given as shown in following tables. Table 1 shows comparison of modifications of RoPE methods for Falcon-7b and LLaMA-2-7b on WiKiText dataset, and table 2 shows comparison of modifications of RoPE methods for LLaMA-2-7b-chat model on several datasets.Table 1Table 2

[0082] The proposed solution can be applied to solve many tasks in future by integrating it into a wide variety of models, and improvements of quality metrics such as accuracy, perplexity and others, are expected to achieve.

[0083] The above is an introduction to the embodiments of the method, and embodiments of an apparatus is described below.

[0084] FIG. 8 is an example of an apparatus 10 proposed by the present application. The apparatus 10 may include corresponding modules or units configured to implement methods and / or embodiments described herein. In some embodiments, the apparatus 10 includes a processing module 11 and a communication module 12. Optionally, the apparatus 10 may further include a storage module 13 configured to store computer program code (or instructions) and / or data. The processing module 11 may be a processor. The communication module 12 may include a receiving module and / or a transmitting module'. The receiving module and / or the transmitting module may be a transmitter and a receiver respectively. The storage module 13 may be a memory.

[0085] In some implementations, a function of the apparatus 10 may be implemented by one or more processors. Specifically, the processor may include a modem chip, or a system on chip SoC chip or a system in package (SIP) that includes a modem core. A function of thecommunication module 12 may be implemented by a transceiver circuit.

[0086] In some implementations, when the apparatus 10 is a circuit or a chip in a terminal, for example, a modem chip, a system on chip SoC chip or a system in package (SIP) chip that includes a modem core, a function of the processing module 11 may be implemented by a circuit system that is in the chip and that includes one or more processors or processor cores. A function of the communication module 12 may be implemented by an interface circuit or a data transceiver circuit on the foregoing chip.

[0087] The modules in the apparatus 10 may be logical or functional. Each function may correspond to one functional module, or two or more functions may be integrated into one functional module. In actual implementation, all or some of the modules may be integrated into one physical entity, or may be distributed in different physical entities. In addition, the foregoing functional modules may be implemented in a form of hardware, may be implemented in a form of software, or may be implemented in a form of a combination of hardware and software. Whether a function is performed in a form of hardware or software depends on particular applications and design constraint conditions of the technical solutions. A person skilled in the art may use different methods to implement the described functions for each particular application, but it should not be considered that the implementation goes beyond the scope of this application.

[0088] In an example, a functional module in any one of the foregoing apparatus may be configured as one or more integrated circuits for implementing the methods disclosed herein, for example, one or more application-specific integrated circuits (application-specific integrated circuits, ASICs), one or more central processing modules (central processing modules, CPUs), one or more microprocessors (microcontroller modules, MCUs), one or more digital signal processors (digital signal processors, DSP), one or more field programmable gate arrays (field programmable gate arrays, FPGAs), or a combination of at least two of these integrated circuit forms.

[0089] In an example, the storage module 13 may include a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, and / or a register.

[0090] A processor, a processor system, an application processor, a baseband processor, a processor circuit, or a processor core may be collectively referred to as a processor. The processor may include one or a combination of a central processing unit (CPU), a digital signal processorPCI7RU2025 / 000044(DSP), a microprocessor (microprocessor unit, MPU), a microcontroller (microcontroller unit, MCU), a graphics processing unit (GPU), a field programmable gate array (FPGA), an artificial intelligence processor (Al processor), or a neural network processing unit (NPU).

[0091] Memory or a storage module may include one or more of the following storage media: a random access memory (RAM), a static random access memory (static RAM, SRAM), a dynamic random access memory (dynamic RAM, DRAM), a phase-change memory (PCM), a resistive random access memory (resistive RAM, ReRAM), a magnetoresistive random access memory (magnetoresistive RAM, MRAM), a ferroelectric random access memory (ferroelectric RAM, FRAM), a cache, a register, a read-only memory (ROM), a flash memory (flash memory), an erasable programmable read-only memory (erasable programmable ROM, EPROM), a hard disk, and the like. In an example, computer program instructions used to execute embodiments may be stored in a non-volatile memory, for example, at least a part of a memory or storage module (for example, one or more of a ROM, a flash memory, an EPROM, or a hard disk). When an apparatus runs, a part or all of corresponding computer program instructions may be loaded to a memory that has a higher transmission speed with the processor, for example, at least a part of a memory or a storage module (for example, one or more of a RAM, an SRAM, a DRAM, a PCM, a RERAM, an MRAM, a FRAM, a cache, or a register), so that the processor executes the computer program instructions to perform the steps in the method embodiments disclosed herein.

[0092] An embodiment of the present application further provides a computer storage medium, and the computer storage medium may store one or more instructions for executing any of the foregoing methods.

[0093] An embodiment of the present application further provides a computer program product, and the computer program product may store one or more instructions for executing any of the foregoing methods.

[0094] In the present disclosure, the terms “a”, “an” and “one” are defined to mean “at least one”, that is, these terms do not exclude a plural number of items, unless stated otherwise.

[0095] In the present disclosure, terms such as “substantially”,, “generally” and “about”, which modify a value, condition or characteristic of a feature of an example embodiment, should be understood to mean that the value, condition or characteristic is defined within tolerances that are acceptable for the proper operation of this example embodiment for its intended application.

[0096] In the present disclosure, unless stated otherwise, the terms “connected” and “coupled”, and derivatives and variants thereof, refer herein to any structural or functional connection or coupling, either direct or indirect, between two or more elements. For example, the connection or coupling between the elements can be acoustical, mechanical, optical, electrical, thermal, logical, or any combinations thereof.

[0097] In the present disclosure, the expression “based on” is intended to mean “based at least partly on”, that is, this expression can mean “based solely on” or “based partially on”, and so should not be interpreted in a limited manner. More particularly, the expression “based on” could also be understood as meaning “depending on”, “representative of’, “indicative of’, “associated with” or similar expressions.

[0098] In the present disclosure, "At least one" means one or more, and "a plurality of' means two or more. The term "and / or" describes an association relationship of associated objects, and indicates that three relationships may exist. For example, A and / or B may indicate the following three cases: Only A exists, both A and B exist, and only B exists, where A and B may be singular or plural. The character 7" indicates an "or" relationship between associated objects. "At least one of the following items (pieces)" or a similar expression thereof indicates any combination of these items, including a single item (piece) or any combination of a plurality of items (pieces). For example, "at least one of A, B, or,C" includes: only A; only B; only C; A and B; A and C; B and C; or A, B, and C, and "at least one of A, B, and C" may also be understood as including: only A; only B; only C; A and B; A and C; B and C; or A, B, and C. In addition, unless otherwise specified, ordinal numbers such as "first" and "second" in embodiments of this application are used to distinguish between a plurality of objects, and are not used to limit a sequence, a time sequence, priorities, or importance of the plurality of objects.

[0099] A person skilled in the art should understand that embodiments of this application may be provided as a method, an apparatus (or system), computer-readable storage medium, or a computer program product. Therefore, this application may use a form of a hardware-only embodiment, a software-only embodiment, or an embodiment vith a combination of software and hardware. Moreover, this application may use a form of a computer program product that is implemented on one or more computer-usable storage media (including but not limited to a disk memory, an optical memory, and the like) that include computer-usable program code.

[0100] This application is described with reference to the flowcharts and / or block diagrams of the method, the device (system), and the computer program product according to this application. It should be understood that computer program instructions may be used to implement each process and / or each block in the flowcharts and / or the block diagrams and a combination of a process and / or a block in the flowcharts and / or the block diagrams. The computer program instructions may be provided for a general-purpose computer, a dedicated computer, an embedded processor, or a processor of another programmable data processing device to generate a machine, so that the instructions executed by the computer or the processor of the another programmable data processing device generate an apparatus for implementing a specific function in one or more procedures in the flowcharts and / or in one or more blocks in the block diagrams.[ooioi] The computer program instructions may alternatively be stored in a computer-readable memory that can indicate a computer or' another programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate an artifact that includes an instruction apparatus. The instruction apparatus implements a specific function in one or more procedures in the flowcharts and / or in one or more blocks in the block diagrams.

[0102] The computer program instructions may alternatively be loaded onto a computer or another programmable data processing device, so that a series of operations and steps are performed on the computer or the another programmable device, so that computer-implemented processing is generated. Therefore, the instructions executed on the computer or the another programmable device provide steps for implementing a specific function in one or more procedures in the flowcharts and / or in one or more blocks in the block diagrams.

[0103] A person of ordinary skill in the art will be aware that, in combination with the examples described in the embodiments disclosed in this specification, units and algorithm steps may be implemented by using electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed by using hardware or software depends on particular applications and design constraint conditions of the technical solutions. A person skilled in the art may use different methods to implement the described functions for each particular application, but it should not be considered that the embodiment goes beyond the scope of this application.

[0104] It would be understood by a person skilled in the art that, for the purpose of convenience and brevity, in a detailed working process of the foregoing system, apparatus, and unit, reference may be made to a corresponding process in the foregoing method embodiments, and details are not described herein again.

[0105] In the several embodiments provided in this application, the disclosed system, apparatus, and method may be implemented in other manners.' For example, the described apparatus embodiment is merely an example. For example, the module division is a logical function division and other methods of division may be used in an actual embodiment. For example, a plurality of units or components may be combined or integrated into another system, or some features may be ignored or not performed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections may be implemented using various communication interfaces. The indirect couplings or communication connections between the apparatuses or units may be implemented in electronic, mechanical, or other forms.

[0106] In addition, function modules in the embodiments of this application may be integrated into one processing module, each of the modules may exist alone physically, or two or more modules may be integrated into one module.

[0107] When the functions are implemented in the form of a software functional module and sold or used as an independent product, the functions may be stored in a computer-readable storage medium. The technical solutions of this application may be implemented in the form of a software product. The software product is stored in a storage medium, and includes several instructions for instructing a computer device (which may be a personal computer, a server, a network device, or the like) to perform all or some of the steps of the methods described in the embodiments of this application. The foregoing storage medium includes any medium that can store program code, such as a USB flash drive, a removable hard disk, a ROM, a RAM, a magnetic disk, an optical disc or the like.

[0108] The modules described as separate parts may be or may not be physically separate, and parts displayed as modules may be or may not be physical modules, may be located in one position, or may be distributed on a plurality of network modules. Some or all of the modules may be selected based on actual requirements to achieve the objectives of the solutions of the embodiments. In addition, functional modules in the embodiments of this application may be integrated into oneprocessing module, or each of the modules may exist alone physically, or two or more modules are integrated into one module.

[0109] The foregoing descriptions are merely specific implementations of this application, but are not intended to limit the protection scope of this application. Any variation or replacement readily figured out by a person skilled in the art within the technical scope disclosed in this application shall fall within the protection scope of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

Claims

CLAIMSWhat is claimed is:

1. A method for rotary position encoding, comprising:obtaining a first vector of position information, wherein the position information is represented based on a first base value of a frequency, and the first base value is represented in a complex domain; andimplementing rotary position encoding based on the first vector.

2. The method according to claim 1, wherein the first base value is represented based on a second base value of the frequency and a scale factor, and the second base value is represented in a real domain.

3. The method according to claim 1 or 2, wherein the first base value is defined as:b' = b * sa+ iawherein o>= d is an embedding dimension, a is a multiple of reciprocal of b, b' isthe first base value, b is the second base value, s is a scale factor, and i is an imaginary unit.

44. The method according to claim 3, wherein a=5. The method according to claim 3 or 4, wherein a rotation matrix Re:<Pimused in the rotary position encoding is represented as:&wherein ch is a hyper index of a positioncorresponding to the position information in the first vector, mdj is a rotation angle for position indicated6. The method according to claim 5, wherein a length of the first vector is stretched by a stretching value ch Pj.

7. An encoding apparatus, comprising:a receiving module, configured to obtain a first vector of position information, wherein the position information is represented based on a first base value of a frequency, and the first base value is represented in a complex domain; andPC17RU2025 / 000044a processing module, configured to implement rotary position encoding based on the first vector.

8. The encoding apparatus according to claim 7, wherein the first base value is represented based on a second base value of the frequency and a scale factor, and the second base value is represented in a real domain.

9. The encoding apparatus according to claim 7 or 8, wherein the first base value is defined as:b' = b * sa+ iawherein co= d is an embedding dimension, a is a multiple of reciprocal of b, b' is thefirst base value, b is the second base value, s is the scale factor, and i is an imaginary unit.

410. The encoding apparatus according to claim 9, wherein a=11. The encoding apparatus according to claim 9 or 10, wherein a rotation matrix RQi<Pimused in the rotary position encoding is represented as:<wherein ch is a hyper index of a positioncorresponding to the position information in the first vector, mOj is a rotation angle for position indicated<12. The encoding apparatus according to claim 11, wherein a length of the first vector is stretched by a stretching value ch pj.t■ •13. An encoding apparatus, configured to perform the methdd according to any one of claims 1 to 6.

14. The encoding apparatus according to claim 13, wherein comprising:an interface circuit, configured to obtain a first vector of position information, wherein the position information is represented based on a first base value of a frequency, and the first base value is represented in a complex domain; andone or more processors, configured to implement rotary position encoding based on the first vector.

15. The encoding apparatus according to claim 14, wherein the interface circuit comprisesone or more transceivers.

16. An apparatus, comprising:one or more processors; anda memory storing instructions which, when executed by the one or more processors, cause the apparatus to perform the method of any one of claims 1 to 6.

17. A computer-readable storage medium having instructions stored thereon which, when executed by an apparatus, cause the apparatus to perform the method of any one of 1 to 6.

18. A computer program product storing instructions which, when executed, cause an apparatus to perform the method of any one of claims 1 to 6.

Images