Index prefix compression processing method and apparatus, computer device, readable storage medium, and program product
By generating and replacing prefix symbols, the storage overhead problem of traditional B+ tree indexes in similar prefix datasets is solved, achieving efficient index compression and query optimization, adapting to data changes, and improving system performance and scalability.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- CHINA TELECOM CLOUD TECH CO LTD
- Filing Date
- 2025-11-17
- Publication Date
- 2026-06-04
Smart Images
Figure CN2025135300_04062026_PF_FP_ABST
Abstract
Description
Index prefix compression processing methods, apparatus, computer equipment, readable storage media, and program products
[0001] Related applications
[0002] This application claims priority to Chinese patent application filed on November 26, 2024, application number 2024116993287, entitled "Index Prefix Compression Processing Method, Apparatus, Computer Equipment, Readable Storage Medium and Program Product", the entire contents of which are incorporated herein by reference. Technical Field
[0003] This application relates to the field of database technology, and in particular to an index prefix compression processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology
[0004] In the context of data-intensive applications, one of the core challenges of a database management system (DBMS) is how to effectively manage the ever-increasing storage demands while ensuring efficient query response.
[0005] Currently, B+ trees, as a widely used indexing structure, occupy a central position in large-scale data storage and retrieval due to their excellent query performance and range scan capabilities. However, in scenarios containing a large number of datasets with similar prefixes, especially in environments with limited memory and disk resources, traditional B+ tree indexes may lead to significant storage overhead. Summary of the Invention
[0006] Therefore, it is necessary to provide an index prefix compression processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the above-mentioned technical problems.
[0007] Firstly, this application provides an index prefix compression processing method, including:
[0008] The symbol generation model generates prefix symbols to simplify the representation of the corresponding repeated prefix parts based on the repeated prefix parts of each original index; wherein, the symbol generation model generates the prefix symbols corresponding to the original index based on the usage of historical indexes containing historical prefix symbols and / or the usage of index samples containing prefix symbol samples.
[0009] Register each prefix symbol corresponding to each repeating prefix part in the prefix symbol table;
[0010] For the target index, based on the prefix symbols registered in the prefix symbol table, replace the duplicate prefix parts of the target index with the corresponding prefix symbols.
[0011] In one embodiment, a symbol generation model is used to generate a prefix symbol for simplifying the representation of the corresponding repeated prefix part based on the repeated prefix part of each original index. This includes: if a time period condition is met, generating a prefix symbol for simplifying the representation of the corresponding repeated prefix part based on the repeated prefix part of each original index using the symbol generation model.
[0012] In one embodiment, after generating prefix symbols for simplifying the representation of the corresponding repeated prefix parts based on the repeated prefix parts of each original index using a symbol generation model, the method further includes: determining the set of original indexes for which prefix symbols need to be regenerated based on the usage of each prefix symbol; regenerating prefix symbols for simplifying the representation of the corresponding repeated prefix parts based on the repeated prefix parts of each original index in the original index set using a symbol generation model; and registering each regenerated prefix symbol corresponding to the repeated prefix parts of each original index in the original index set in a prefix symbol table.
[0013] In one embodiment, a symbol generation model is used to generate a prefix symbol to simplify the representation of the corresponding repeated prefix portion based on the repeated prefix portion of each original index. This includes: based on the index insertion operation, and under the condition of satisfying resource compression, generating a prefix symbol to simplify the representation of the corresponding repeated prefix portion based on the repeated prefix portion of each original index using the symbol generation model; wherein each original index includes the inserted index and the indexes that existed before insertion in the index page; for the target index, the repeated prefix portion of the target index is replaced with the corresponding prefix symbol based on the prefix symbols registered in the prefix symbol table, including: for the inserted index, the repeated prefix portion of the inserted index is replaced with the corresponding prefix symbol based on the prefix symbols registered in the prefix symbol table.
[0014] In one embodiment, the method further includes: if the index page meets the page fill threshold and the released space of the index page is insufficient, then it is determined that the resource compression condition is met based on the index insertion operation.
[0015] In one embodiment, the compressed indexes in the index page are provided with an identifier bit; the identifier bit is used to indicate that the index has been compressed; the compressed indexes include the inserted indexes and the updated indexes in the existing indexes before insertion; the uncompressed indexes in the index page include the unupdated indexes in the existing indexes before insertion.
[0016] Secondly, this application also provides an index prefix compression processing apparatus, comprising:
[0017] The symbol generation model generates prefix symbols to simplify the representation of the corresponding repeated prefix parts based on the repeated prefix parts of each original index. Specifically, the symbol generation model generates prefix symbols corresponding to the original index based on the usage of historical indexes containing historical prefix symbols and / or the usage of index samples containing prefix symbol samples. Each prefix symbol corresponding to each repeated prefix part is registered in the prefix symbol table. For the target index, the repeated prefix parts of the target index are replaced with the corresponding prefix symbols based on the prefix symbols registered in the prefix symbol table.
[0018] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0019] The symbol generation model generates prefix symbols to simplify the representation of the corresponding repeated prefix parts based on the repeated prefix parts of each original index. Specifically, the symbol generation model generates prefix symbols corresponding to the original index based on the usage of historical indexes containing historical prefix symbols and / or the usage of index samples containing prefix symbol samples. Each prefix symbol corresponding to each repeated prefix part is registered in the prefix symbol table. For the target index, the repeated prefix parts of the target index are replaced with the corresponding prefix symbols based on the prefix symbols registered in the prefix symbol table.
[0020] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:
[0021] The symbol generation model generates prefix symbols to simplify the representation of the corresponding repeated prefix parts based on the repeated prefix parts of each original index. Specifically, the symbol generation model generates prefix symbols corresponding to the original index based on the usage of historical indexes containing historical prefix symbols and / or the usage of index samples containing prefix symbol samples. Each prefix symbol corresponding to each repeated prefix part is registered in the prefix symbol table. For the target index, the repeated prefix parts of the target index are replaced with the corresponding prefix symbols based on the prefix symbols registered in the prefix symbol table.
[0022] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:
[0023] The symbol generation model generates prefix symbols to simplify the representation of the corresponding repeated prefix parts based on the repeated prefix parts of each original index. Specifically, the symbol generation model generates prefix symbols corresponding to the original index based on the usage of historical indexes containing historical prefix symbols and / or the usage of index samples containing prefix symbol samples. Each prefix symbol corresponding to each repeated prefix part is registered in the prefix symbol table. For the target index, the repeated prefix parts of the target index are replaced with the corresponding prefix symbols based on the prefix symbols registered in the prefix symbol table.
[0024] Details of one or more embodiments of this application are set forth in the following drawings and description. Other features and advantages of this application will become apparent from the specification, drawings, and claims. Attached Figure Description
[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the disclosed drawings without creative effort.
[0026] Figure 1 is an application environment diagram of the index prefix compression processing method according to one or more embodiments;
[0027] Figure 2 is a flowchart illustrating the index prefix compression processing method according to one or more embodiments;
[0028] Figure 3 is a schematic diagram comparing the effects of the index prefix compression processing methods according to one or more embodiments;
[0029] Figure 4 is a structural block diagram of the index prefix compression processing apparatus according to one or more embodiments;
[0030] Figure 5 is an internal structural diagram of a computer device according to one or more embodiments. Detailed Implementation
[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0032] The index prefix compression processing method provided in this application embodiment can be applied to the application environment shown in Figure 1. This application environment may include a terminal and a server, and the terminal can communicate with the server via a network. A data storage system can store the data that the server needs to process. The data storage system can be integrated on the server or placed on the cloud or other network servers. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.
[0033] In one exemplary embodiment, as shown in FIG2, an index prefix compression processing method is provided, which can be applied to the server in FIG1. The method may include the following steps:
[0034] Step S201: Based on the repeated prefix part of each original index, generate a prefix symbol to simplify the representation of the corresponding repeated prefix part using the symbol generation model.
[0035] The original index can be the original index contained in the index page before compression. The repeating prefix of the original index can be the repeating prefix of each original index in the index page, and the repeating prefix can be the entire original index. For example, if the original index in the index page includes several indexes str012305, then the repeating prefix can be the entire index str012305.
[0036] The prefix symbol is used to simplify and replace the corresponding repeating prefix part, and can contain characters, numbers, etc. As an example, the prefix symbol for index str012305 could be SP1, etc.
[0037] In this step, a symbol generation model can be used to generate prefix symbols to simplify the representation of the corresponding repeating prefix parts of each original index contained in the index page. This can reduce the database index space.
[0038] The symbol generation model generates prefix symbols corresponding to the original index based on the usage of historical indexes containing historical prefix symbols and / or the usage of index samples containing prefix symbol samples. Historical prefix symbols can include prefix symbols used on other index pages or prefix symbols previously used on that index page. The historical index is the index corresponding to the historical prefix symbol, and its usage can be based on the usage of these historical prefix symbols by relevant users, such as the number of times they were used or whether they are recommended. Prefix symbol samples can include prefix symbol samples set by relevant users for specific index samples. These can be samples that have actually been used or samples set according to the user's own usage habits. When setting the prefix symbol samples for index samples, the usage of these prefix symbol samples can be set simultaneously, such as whether they are recommended. Based on this, the symbol generation model (which can be an artificial intelligence model, neural network, etc.) can be trained using the usage of historical indexes containing historical prefix symbols and / or the usage of index samples containing prefix symbol samples. The trained symbol generation model then generates prefix symbols to simplify the representation of the corresponding repeated prefix parts based on the repeated prefix parts of each original index in the index page.
[0039] Step S202: Register each prefix symbol corresponding to each repeated prefix part in the prefix symbol table.
[0040] In this step, the prefix symbol table can be set in the index page, and the prefix symbol corresponding to each repeated prefix part can be registered in the prefix symbol table.
[0041] Step S203: For the target index, based on the prefix symbols registered in the prefix symbol table, replace the duplicate prefix part of the target index with the corresponding prefix symbol.
[0042] In this step, for the target index that requires index prefix compression, the repeated prefix portion of the target index can be replaced with the corresponding prefix symbol based on the prefix symbols registered in the prefix symbol table, and the prefix-compressed target index is recorded in the index page. Therefore, when writing the index, the repeated prefix portion can be replaced with prefix symbol references based on the prefix symbol table, and during reading, it can be parsed in reverse.
[0043] Figure 3 shows a comparison of the index page before and after compression. The target index can include all the indexes contained in the index page before compression. By mapping the repeated prefix parts to prefix symbols used to simplify the representation of the corresponding repeated prefix parts through the symbol generation model, the prefix symbol corresponding to each repeated prefix part is registered in the prefix symbol table set in the index page. For all the indexes contained in the index page before compression, their repeated prefix parts can be replaced with the corresponding prefix symbols. The indexes after prefix compression can still be arranged in a certain order in the index page, and can be displayed in a certain order following the prefix symbol table. Therefore, based on the intelligent shared prefix storage mechanism, through the integration and function of the prefix symbol table, frequently occurring prefixes can be stored in a static form. By centrally storing repeated prefix information and retaining it only once, while storing the rest as independent suffixes, index space is greatly saved and parsing time is reduced. This not only reduces duplicate data but also simplifies the query process and improves efficiency. Furthermore, it can be combined with deep learning to generate prefix symbols suitable for each index in the current index page, improving compression efficiency and reducing space occupation. Based on the sorting characteristics of the index, by dividing the data into static and dynamic forms, the database index space can be compressed while maintaining or improving the timeliness of query processing.
[0044] The index prefix compression method in this embodiment generates prefix symbols to simplify the representation of the repeated prefix parts of each original index based on a symbol generation model. The symbol generation model generates prefix symbols corresponding to the original index based on the usage of historical indexes containing historical prefix symbols and / or the usage of index samples containing prefix symbol samples. Each prefix symbol corresponding to each repeated prefix part is registered in the prefix symbol table. For the target index, the repeated prefix parts of the target index are replaced with the corresponding prefix symbols according to the prefix symbols registered in the prefix symbol table. This method can intelligently generate prefix symbols to simplify the representation of the corresponding repeated prefix parts of each original index, thereby achieving intelligent compression of the index prefix, improving compression efficiency, reducing its space occupation, and reducing the storage overhead of the index.
[0045] In one exemplary embodiment, step S201, which generates a prefix symbol for simplifying the representation of the corresponding repeated prefix portion based on the repeated prefix portion of each original index using a symbol generation model, may include:
[0046] If the time period condition is met, the symbol generation model generates a prefix symbol to simplify the representation of the corresponding repeated prefix part based on the repeated prefix part of each original index.
[0047] In this embodiment, the time period condition can be a day, a week, a month, or other time period set by the relevant user. If the current time meets the time period condition, a symbol generation model can be used to generate a prefix symbol to simplify the representation of the corresponding repeated prefix part based on the repeated prefix part of each original index. The original indexes can include all the original indexes contained in the index page at the current time. This allows for the compression of each original index contained in the index page according to a certain time period, continuously optimizing the simplified representation of the corresponding repeated prefix part of each original index in the index page.
[0048] In one exemplary embodiment, after generating prefix symbols for simplifying the representation of the corresponding repeated prefix portions based on the repeated prefix portions of each original index using the symbol generation model in step S201, the method may further include:
[0049] Based on the usage of each prefix symbol, determine the original index set that needs to be regenerated with prefix symbols; using the symbol generation model, regenerate prefix symbols to simplify the representation of the corresponding repeated prefix parts based on the repeated prefix parts of each original index in the original index set; register each regenerated prefix symbol corresponding to the repeated prefix parts of each original index in the original index set in the prefix symbol table.
[0050] In this embodiment, the set of original indexes in the index page that require regenerated prefix symbols can be determined based on the usage of the prefix symbols corresponding to each original index in the index page, such as usage frequency and whether their use is recommended. The set of original indexes requiring regenerated prefix symbols may include, for example, original indexes whose usage frequency does not meet the requirement. In this case, prefix symbols can be regenerated for them according to the user's usage habits. The symbol generation model regenerates prefix symbols to simplify the representation of the corresponding repeated prefix parts of each original index in the original index set. These regenerated prefix symbols should be different from the prefix symbols corresponding to the original indexes whose usage frequency does not meet the requirement; if they are the same, they can be regenerated again. After regenerating the prefix symbols, each regenerated prefix symbol corresponding to the repeated prefix parts of each original index in the original index set can be registered in the prefix symbol table. This allows for the regeneration of prefix symbols when their usage does not meet the user's requirements, improving the adaptability of the prefix symbols.
[0051] In one exemplary embodiment, step S201, which generates a prefix symbol to simplify the representation of the corresponding repeated prefix portion based on the repeated prefix portion of each original index using a symbol generation model, may include: generating a prefix symbol to simplify the representation of the corresponding repeated prefix portion based on the repeated prefix portion of each original index using a symbol generation model, provided that the resource compression conditions are met, according to the index insertion operation; wherein each original index includes the inserted index and the index that existed before insertion in the index page.
[0052] Compression logic can be intelligently inserted during the transaction commit phase, when the index is converted from an in-memory structure to a persistent structure. This process not only considers the atomicity of transactions but also utilizes transaction locking mechanisms to ensure data consistency and integrity during the conversion and compression processes.
[0053] In this embodiment, based on the index insertion operation, and provided that compression resource conditions are met (e.g., one or more compression resource conditions such as current system load and memory status are satisfied), a symbol generation model generates a prefix symbol to simplify the representation of the corresponding repeated prefix portion based on the repeated prefix portion of each original index. At this time, each original index can include the inserted index and existing indexes in the index page (or index page) before insertion. That is, when inserting an index, if the current compression resource conditions are met, compression processing can be triggered on all existing indexes in the index page before insertion, as well as the newly inserted index. Therefore, an aggressive or conservative compression strategy can be adopted based on the compression resource conditions. For example, a more aggressive compression strategy can be adopted when resources are sufficient and index updates are infrequent, while a conservative strategy can be adopted to reduce CPU and I / O overhead and ensure overall system performance stability.
[0054] Compression can also be performed outside of the index management operation (SMO), avoiding the holding of expensive index locks and multiple page locks, reducing blocking of concurrent operations, and improving the system's concurrency processing capabilities. Furthermore, compression decisions are made only at the page level, without relying on global information, simplifying the complexity of the compression algorithm.
[0055] Based on this, step S203, which involves replacing the duplicate prefix portion of the target index with the corresponding prefix symbol according to the prefix symbol registered in the prefix symbol table, may include:
[0056] For the inserted index, based on the prefix symbols registered in the prefix symbol table, replace the repeated prefix part of the inserted index with the corresponding prefix symbol.
[0057] In this embodiment, after triggering the compression process of all existing indexes and newly inserted indexes in the index page before insertion, the duplicate prefix part of the inserted index can be replaced with the corresponding prefix symbol according to the prefix symbol registered in the prefix symbol table.
[0058] In one exemplary embodiment, the method of this application may further include the following steps: if the index page meets the page fill threshold and the released space of the index page is insufficient, then it is determined that the resource compression condition is met according to the index insertion operation.
[0059] In this embodiment, a page fill threshold can be set. When the index page meets the page fill threshold (close to full load) and cannot release enough space through methods such as page reorganization (insufficient free space in the index page), the resource compression condition is determined to be met. Only then is batch compression of the entire index page's data triggered. This embodiment effectively distributes the overhead of compression operations, avoiding the high cost of compression with every insertion operation. Batch processing reduces frequent input / output (IO) operations, smoothing the system resource usage curve. Especially in high-concurrency write scenarios, it significantly improves system stability and response speed.
[0060] In one exemplary embodiment, the compressed index in the index page is provided with an identifier bit; the identifier bit is used to indicate that the index has been compressed; the compressed index includes the inserted index and the updated index among the existing indexes before insertion; the uncompressed index in the index page includes the unupdated index among the existing indexes before insertion.
[0061] In this embodiment, a flag can be set for compressed indexes in the index page. The flag indicates that the index in the index page has been compressed (this can be done by adding specific bits to the header of each index to indicate whether it has undergone compression), thus quickly distinguishing between compressed and uncompressed indexes and facilitating on-demand processing by the system. Compressed indexes can include newly inserted indexes and updated indexes from existing indexes before insertion. Uncompressed indexes in the index page can include unupdated indexes from existing indexes before insertion, which can be understood as indexes from older versions of the database.
[0062] In this embodiment, a universal flag bit ensures data consistency during database upgrades or migrations, and the flag bit fully considers forward and backward compatibility. Specifically, for indexes in older database versions, newly added flag bits can be disabled by default to ensure existing query logic remains unaffected. For newly inserted or updated indexes, flag bits can be set as needed, allowing a gradual transition to the new storage model without damaging historical data. Thus, compressed and uncompressed indexes can coexist on the same index page. The flag bit system can dynamically identify and handle different types of indexes. This hybrid storage model fully utilizes page space, especially in scenarios with frequent data updates. The strategy of coexisting old and new indexes helps to smoothly transition to a more efficient compressed storage format. Based on this, during query operations, the system can quickly distinguish index types based on flag bits and select appropriate decompression paths. Update operations are dynamically adjusted according to the index status, potentially involving record decompression, updating, and recompression, or performing updates directly in compressed state to minimize resource consumption. Through index design and a hybrid storage model, the database system can achieve efficient storage space utilization and flexible storage strategy adjustments without sacrificing query efficiency. This not only enhances the system's scalability but also provides strong support for meeting ever-changing data processing needs.
[0063] In some applications, efficient decompression logic and data access optimization can be implemented to ensure data access performance. Specifically, a flag can be used to indicate whether an index is compressed. To ensure seamless index access, adjustments can be made to enable the identification and correct handling of compressed indexes. This can include introducing parsing of compressed metadata into the decompression logic and processing of compressed fields when obtaining index offsets, ensuring accurate index location even in compressed states. The optimal decompression path can be selected based on query conditions and operation type. For simple read operations, only the necessary fields can be decompressed; for complex queries or update operations, the entire index may need to be decompressed. Flexible strategies can reduce unnecessary data decompression, significantly improving processing speed. To address potential loss of column attribute information in compressed indexes, a metadata mapping table corresponding to the compressed index can be maintained. This allows for rapid recovery of column attributes during comparisons or transformations, ensuring index integrity and operational accuracy. Furthermore, based on the context of the current operation (such as query conditions and the current stage of the execution plan), it can dynamically decide whether to immediately decompress the index, to what extent to decompress it, or to directly perform comparison operations in the compressed domain. This minimizes the overhead of data decompression while ensuring data consistency. Therefore, it ensures that when processing compressed data, both high-speed data access and efficient utilization of storage resources are maintained, providing strong performance support for modern database management systems.
[0064] Overall, the index prefix compression method in this application, through a semi-static prefix strategy and symbol table design, achieves efficient reuse and storage compression of prefix information, significantly reducing the storage space required for the index, especially when processing highly repetitive data. This reduces hardware resource requirements and maintenance costs, thereby improving storage efficiency and resource utilization. It can also intelligently adjust the compression timing and method dynamically based on system load and other information, avoiding negative impacts on system performance while ensuring efficient data access. This facilitates optimization of the entire decompression logic chain in applications, ensuring a seamless data access experience and reducing query latency. Secondly, the introduction of flag bits and hybrid storage modes not only ensures a smooth transition and compatibility between old and new data formats but also improves the system's scalability and flexibility, allowing free switching between different record formats to adapt to diverse business scenarios and data processing needs. Multi-level decompression strategies and intelligent selection mechanisms can be applied, dynamically adjusting the decompression level according to query requirements, reducing unnecessary data processing, improving query performance, and resolving column attribute missing issues to ensure the accuracy and integrity of data processing. Therefore, the index prefix compression method of this application can provide technical support for database management and data-intensive applications, and solve the storage and performance bottleneck problems encountered when processing large-scale data.
[0065] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0066] Based on the same inventive concept, this application also provides an index prefix compression processing apparatus for implementing the index prefix compression processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more index prefix compression processing apparatus embodiments provided below can be found in the limitations of the index prefix compression processing method described above, and will not be repeated here.
[0067] In one exemplary embodiment, as shown in FIG4, an index prefix compression processing apparatus is provided, the apparatus 400 including:
[0068] The symbol generation module 401 is used to generate a prefix symbol for simplifying the representation of the corresponding repeated prefix part based on the repeated prefix part of each original index through a symbol generation model; wherein, the symbol generation model generates the prefix symbol corresponding to the original index based on the usage of historical indexes containing historical prefix symbols and / or the usage of index samples containing prefix symbol samples.
[0069] Symbol registration module 402 is used to register each of the prefix symbols corresponding to each of the repeated prefix portions in the prefix symbol table;
[0070] The prefix replacement module 403 is used to replace the repeated prefix portion of the target index with the corresponding prefix symbol according to the prefix symbol registered in the prefix symbol table.
[0071] In one exemplary embodiment, the symbol generation module 401 is configured to generate a prefix symbol for simplifying the representation of the corresponding repeated prefix portion based on the repeated prefix portion of each original index, if the time period condition is met.
[0072] In one exemplary embodiment, the symbol generation module 401 is further configured to determine the original index set that needs to be regenerated based on the usage of each of the prefix symbols; regenerate prefix symbols for simplifying the representation of the corresponding repeated prefix parts based on the repeated prefix parts of each original index in the original index set using a symbol generation model; and register each of the regenerated prefix symbols corresponding to the repeated prefix parts of each original index in the original index set in the prefix symbol table.
[0073] In one exemplary embodiment, the symbol generation module 401 is configured to, based on the index insertion operation and under the condition of compressing resources, generate a prefix symbol for simplifying the representation of the corresponding repeated prefix portion of each original index using a symbol generation model; wherein, each original index includes the inserted index and the indexes that existed before insertion in the index page; the prefix replacement module 403 is configured to, for the inserted index, replace the repeated prefix portion of the inserted index with the corresponding prefix symbol according to the prefix symbols registered in the prefix symbol table.
[0074] In one exemplary embodiment, the symbol generation module 401 is further configured to determine that the compressed resource condition is met if the index page meets the page fill threshold and the released space of the index page is insufficient, based on the index insertion operation.
[0075] In one exemplary embodiment, the compressed indexes in the index page are provided with an identifier bit; the identifier bit is used to indicate that the index has been compressed; the compressed indexes include the inserted index and the updated indexes among the existing indexes before insertion; the uncompressed indexes in the index page include the unupdated indexes among the existing indexes before insertion.
[0076] Each module in the aforementioned index prefix compression processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.
[0077] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram is shown in Figure 5. The computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device stores index data. The I / O interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an index prefix compression processing method.
[0078] Those skilled in the art will understand that the structure shown in Figure 5 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or may combine certain components, or may have different component arrangements.
[0079] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.
[0080] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above-described method embodiments.
[0081] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.
[0082] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.
[0083] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.
[0084] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0085] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A method for compressing index prefixes, characterized in that, The method includes: A symbol generation model is used to generate prefix symbols for simplifying the representation of the corresponding repeated prefix parts of each original index. The symbol generation model generates the prefix symbols corresponding to the original index based on the usage of historical indexes containing historical prefix symbols and / or the usage of index samples containing prefix symbol samples. Register each of the prefix symbols corresponding to each of the repeated prefix portions in the prefix symbol table; For the target index, based on the prefix symbols registered in the prefix symbol table, the repeated prefix portion of the target index is replaced with the corresponding prefix symbol.
2. The method according to claim 1, characterized in that, The step of generating prefix symbols for simplifying the representation of the corresponding repeated prefix portions based on the repeated prefix portions of each original index using the symbol generation model includes: If the time period condition is met, a prefix symbol for simplifying the representation of the corresponding repeated prefix part is generated by the symbol generation model based on the repeated prefix part of each original index.
3. The method according to claim 1, characterized in that, After generating prefix symbols for simplified representation of the corresponding repeated prefix portions based on the repeated prefix portions of each original index using a symbol generation model, the method further includes: Based on the usage of each prefix symbol, determine the original set of indexes that need to be regenerated for the prefix symbols; The symbol generation model regenerates prefix symbols for simplified representation of the corresponding repeated prefix parts based on the repeated prefix parts of each original index in the original index set. Register in the prefix symbol table each of the newly generated prefix symbols corresponding to the repeated prefix portions of each original index in the original index set.
4. The method according to claim 1, characterized in that, The step of generating prefix symbols for simplifying the representation of the corresponding repeated prefix portions based on the repeated prefix portions of each original index using the symbol generation model includes: Based on the index insertion operation, and under the condition of resource compression, a symbol generation model is used to generate a prefix symbol to simplify the representation of the corresponding repeated prefix part based on the repeated prefix part of each original index; wherein, each original index includes the inserted index and the index that existed before the insertion in the index page; The step of replacing the repeated prefix portion of the target index with the corresponding prefix symbol according to the prefix symbol registered in the prefix symbol table includes: For the inserted index, the repeated prefix portion of the inserted index is replaced with the corresponding prefix symbol according to the prefix symbol registered in the prefix symbol table.
5. The method according to claim 4, characterized in that, The method further includes: According to the index insertion operation, if the index page meets the page fill threshold and the released space of the index page is insufficient, then the resource compression condition is determined to be met.
6. The method according to claim 4, characterized in that, The compressed indexes in the index page are marked with an identifier; the identifier is used to indicate that the index has been compressed; the compressed indexes include the inserted indexes and the updated indexes among the existing indexes before insertion; the uncompressed indexes in the index page include the unupdated indexes among the existing indexes before insertion.
7. An index prefix compression processing device, characterized in that, The device includes: The symbol generation module is used to generate a prefix symbol for simplifying the representation of the corresponding repeated prefix part based on the repeated prefix part of each original index using a symbol generation model; wherein, the symbol generation model generates the prefix symbol corresponding to the original index based on the usage of historical indexes containing historical prefix symbols and / or the usage of index samples containing prefix symbol samples. A symbol registration module is used to register each of the prefix symbols corresponding to each of the repeated prefix portions in the prefix symbol table; The prefix replacement module is used to replace the repeated prefix portion of the target index with the corresponding prefix symbol based on the prefix symbols registered in the prefix symbol table.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.