Similarity-based search optimization

Abstract

A user device submits a query requesting identification of a type of item provided on an exchange platform. A search is performed on the exchange platform in response to the query to generate an initial list of candidate items. A trained large language model (LLM) generates a list of the pairs of the most similar item listings. The most similar item listing pairs are ranked using an LLM. The LLM which ranks the pairs can be the same as the LLM which generated the list of pairs of similar listing but can also be a different LLM. Based on the ranked list of item listing pairs and the received query, a set of final listings is generated and provided for display to a user device.

Description

Attorney Docket No. 47166-0070W01SIMILARITY-BASED SEARCH OPTIMIZATIONBACKGROUND

[0001] Electronic platforms and marketplaces enable exchange of items, services, or digital components between providers of such items / services and end users desiring such items / services. Such platforms typically include search engines that retrieve listings of items / services / digital components in response to user queries, and ranking algorithms that rank the retrieved listings. The order in which listings are provided can be based on various criteria (e.g., relevance, quality, diversity). Depending on the user query for an item, listed items might need to balance diversity of listings with quality or relevance. It is not desirable to show identical products with little diversity in the response to the search query.SUMMARY

[0002] This specification describes machine learning-based techniques for providing an improved or optimized response to a search query by identifying listings for similar or nearly similar items and providing to a user device a list of items for a specified diversity level. Similarity can be based on identity across one or more attributes of the items / services / digital components listings. Items / services / digital components being identical to within a predetermined similarity of each other can be considered identical and collapsed to a single initial presentation to a user device.

[0003] In performing a search for items on an exchange platform, the search algorithm can be configured to balance the relevance of the search with how diverse the results should be. The methods, systems, and techniques described in this disclosure provide a balance between relevant search results and diverse search results to a given query.

[0004] The methods and systems described in this disclosure provide resource efficient techniques to balance between these two goals. A conventional solution to determining the listings to be provided to a user device is to use the maximal marginal relevance (MMR) criterion which, in effect, is a two-stage maximization function of two similarity calculations. When such an algorithm is applied to a set of candidate listings (e.g., listings generated by a retrieval engine in response to the search query), the ranked ordering of such candidate listings generally requires comparing every candidate listing to other candidate listings. This results in a computational cost that grows quadratically with the size of the set to be ranked. In contrast, the techniques describedAttorney Docket No. 47166-0070W01herein can perform the same function but consume substantially fewer computer resources given that these techniques only require ranked ordering that increases in a linear fashion with the number of returned candidate listings. That’s because the techniques described herein use large language models (LLMs) to extract closest (e.g., most similar) pairs of item listings from amongst listings of items and to rank the closest pairs from most similar to least similar are coupled with the use of a diversity parameter to provide results which are not as similar as nearly identical item listings (as further described throughout this document). Since exchange platforms can include hundreds of millions of item listings, the savings thus produced can be substantial.

[0005] In addition, identifying similar items, and grouping them together improves the quality of the returned item listings to the user. The relevance of the items is maintained and the user can determine how diverse the list of returned items should be. The results of the set of final listings presented to a user device is higher in quality since the results are likely to be highly relevant and provide adequate breadth to be interesting to the user. Machine learning (ML) models can be trained to perform these tasks. LLMs are proficient at leveraging, for instance, image information to determine closest pairs. LLMs are also proficient at ranking the pairs based on their similarity. A balance can be achieved between relevance and diversity by using an additional parameter.

[0006] By leveraging its encapsulated knowledge of the world, a pre-trained LLM is inherently capable of identifying the most similar items and ranking pairs of such items. Those outputs can then be configured to surface to the end user the desired balance between relevance and breadth / diversity of results. Identifying similar items from different sellers and surfacing only one of them allows for a more equitable promotion of quasi-interchangeable sellers over time, to avoid situations where, everything else being equal, the same shop always shows up ahead of others in the ranked list. The methods and techniques described in this disclosure also open up the possibility of flagging questionable sellers who might be "drop-shipping" near-identical items on the platform, thus assisting policy enforcement efforts and contributing to enhance customer trust in the marketplace.

[0007] A computer-implemented method is described. The method includes receiving, from a user device and by an exchange platform, a query requesting identification of a type of item provided on the exchange platform. The method includes performing a search on item listings provided inAttorney Docket No. 47166-0070W01a database of item listings to generate a plurality of candidate item listings responsive to the query. The method includes generating, using a trained large language model (LLM) and based on the plurality of candidate item listings, a plurality of most similar item listing pairs. Generating the plurality of most similar item listing pairs can include, for each candidate item listing among the plurality of candidate item listings, identifying another candidate item listing that has attributes corresponding to attributes of the candidate item listing. Generating the plurality of most similar item listing pairs can also include ranking, using the trained LLM, the plurality of most similar item listing pairs to obtain a ranked list of item pairs. The method includes generating, based on the ranked list of item pairs and the received query, a set of final listings. The method includes providing for display, to an application executing within the user device, the set of final listings that are responsive to the query.

[0008] Performing the search on item listings can include performing an approximate nearest neighbor search on item listings provided in the database of item listings responsive to the query. Identifying another candidate item listing that has attributes corresponding to attributes of the candidate item listing can include, calculating a similarity between the candidate item listing among the plurality of candidate item listings and another candidate item listing of the plurality of candidate item listings and pairing the candidate item listing with another candidate item listing with a greatest calculated similarity. Ranking the plurality of most similar item listing pairs to obtain the ranked list of item pairs can include ranking each item listing pair based on the calculated similarity between the item listing and the other item listing. Generating the set of final listings can include truncating the set of final listings based on a threshold diversity value. The threshold diversity value can be received from the user device. The threshold diversity value can be determined by the exchange platform. The threshold diversity value can be one of a number of item listings, a maximum difference in a similarity metric between two members of an item pair, or a threshold similarity metric. Generating the set of final listings can include, for each of the ranked list of item pairs, recursively removing any remaining pairs which include either item listing of a selected item pair.

[0009] A system is presented for performing the computer-implemented method described herein.Attorney Docket No. 47166-0070W01BRIEF DESCRIPTION OF THE DRAWINGS

[0010] FIG. 1 illustrates the components of the system for optimizing search responses.

[0011] FIG. 2 illustrates the various components and some details of operation of the system of FIG. 1.

[0012] FIG. 3 illustrates an example of an initial set of listings retrieved as relevant for a search.

[0013] FIG. 4 illustrates an example interim set of item listings.

[0014] FIG. 5 illustrates an example set of final item listings.

[0015] FIG. 6 illustrates a process flow for generating the optimized search response.

[0016] FIG. 7 illustrates networked electronic devices.

[0017] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0018] This disclosure relates to methods and systems for providing a response to a user query about a search item which is both relevant to the user query and also minimizes the number of nearly similar items retrieved.

[0019] Conventional techniques for providing relevant results to a search query and also providing some diversity typically scale at least quadratically with the number of items being searched. A conventional solution to the problems described herein is to provide a listing of all items, even if many of the item listing are very nearly similar to each other. A user may wish to receive some set of similar items but have the identical or very nearly similar items presented only as a group or to be presented with a representative sample of such items rather than all the items. A user may also wish to receive some items a little further away from the most relevant responses to the query without having to scroll through multiple pages of candidates.

[0020] The methods, systems, and techniques described herein leverage the capabilities of machine learning (ML) models (e.g., an LLM), to provide a balanced response which is both relevant to the query and also provides a diversity of responses. An ML model is used to create aAttorney Docket No. 47166-0070W01list of pairs of item listings. For each item listing, a mate is identified which is the most similar to that item listing. The ML model ranks the group of pairs from the most similar to the least similar. Based on a diversity parameter, a balanced set of final listings is provided to a user device.

[0021] The invention is described herein in the context of exchange platforms but one skilled in the art will appreciate that the invention can be applied in other search contexts where digital components are provided in response to requested search queries. However, for brevity and ease of description, the disclosure here is provided in the context of exchange platforms and items / services listed on such platform.

[0022] FIG. 1 illustrates the components of a system for optimizing a search process for items, services, or other digital components provided on an exchange platform. The system 100 for search optimization includes an exchange platform 110 and can include user devices 150, and lister devices 155 interacting with the exchange platform 110 via a network 160. The user devices 150 can run applications 150-A. The lister devices 155 can run applications 155-A. In some implementations the user device applications 150-A and the lister device applications 155-A include an application provided by the exchange platform 110 to promote ease of interaction. The exchange platform 110 can include one or more computing devices.

[0023] Users of a user device 150 can use an application 150-A to communicate with the exchange platform 110 to, for example, view listings of items, search for items, obtain items, and / or perform other appropriate tasks related to the exchange platform 110. In some examples, listings of items can include data that is associated with items or services that are for sale on the exchange platform 110. For example, the data can include images, text, multimedia, etc. associated with an item or service that is listed for sale. In some examples, listings of items are not limited to data associated with items or services provided for sale on the exchange platform 110 and can refer to data associated with other digital components that may be provided on the exchange platform 110 or other electronic platforms. Digital components can refer to a discrete unit of digital content or digital information (e.g., a video clip, an audio clip, a multimedia clip, an image, text, or another unit of content). A digital component can electronically be stored in a physical memory device as a single fde or in a collection of fdes, and digital components can take the form of video fdes, audio fdes, multimedia fdes, image fdes or text fdes. The techniques discussed herein can beAttorney Docket No. 47166-0070W01applied to listings of items as well of listings of digital components provided on the exchange platform 110 or other platforms. The listings of digital components can be stored and processed in a manner similar to the listings of items discussed herein.

[0024] The exchange platform 110 includes a retrieval engine 115, a search optimization engine 120, which can include (or interact over a network with) a large language model (LLM) 122, a ranking engine 130, and a listing database (DB) 140. The exchange platform 110 may receive a listing request from a lister device 155. The lister device 155 transmits a listing for an item to the exchange platform 110 to be listed on the exchange platform 110. The exchange platform 110 stores the listing of the item in the listing DB 140. The retrieval engine 115 can be a machine language (ML) based engine that uses an approach to retrieve results for the query from the listing DB 140. In some implementations, the retrieval engine 115 uses a nearest neighbor approach (e.g., an approximate nearest neighbor approach) to retrieving results from the listing DB 140 in response to the query.

[0025] The exchange platform 110 receives a search query from, for example, a user device 150 over a network 160. The search optimization engine 120 on the exchange platform 110 accesses the listing DB 140 to perform an optimized search, as will be explained below, using the LLM 122. In addition, the exchange platform 110 may include a ranking engine 130 for ranking a plurality of item listings.

[0026] FIG. 2 illustrates some details of the system of FIG. 1. The user device 150 submits a query 152 to the exchange platform 110. The exchange platform 110 actuates the retrieval engine and the search optimization engine 120 and provides both engines access to the listing DB 140. The retrieval engine 115 performs a relevance search on the listing DB 140 using some relevance criterion or criteria for the query 152 and identifies candidate item listings 170. In an example, the retrieval engine 115 performs an approximate nearest neighbor search on the listing DB 140 for the query 152 and provides candidate item listings 170. The candidate item listings 170 includes a plurality of candidate item listings relevant to the query 152.

[0027] The search optimization engine 120 then extracts the closest pairs — the most similar pairs of item listings from the candidate item listings 170 — to form an unranked list of candidate item listing pairs 175. In an example, the search optimization engine 120 calculates a similarity betweenAttorney Docket No. 47166-0070W01each item listing with every other item listing in the candidate item listings 170. For each item listing, the other item listing is identified, with which the item listing has the maximum similarity. In some implementations, an LLM 122 is used to identify the pairs from the plurality of candidate item listings relevant to the query 152.

[0028] From the candidate item listing pairs 175, the ranking engine 130 generates a ranked listing of pairs 180, ordered from a most similar pair to a least similar pair. In some implementations, the ranking engine 130 can use an LLM 122 to rank the pairs of item listings. The ranking engine 130 removes all pairs which include a candidate listing which is already present in another pair. The ranking engine 130 the ranked listing pairs 180.

[0029] In some implementations, the search optimization engine 120 uses an LLM 122 to generate the candidate item listing pairs 175 from the candidate item listings 170. In some implementations the ranking engine 130 uses the LLM 122 to rank the candidate item listing pairs to generate the ranked listing pairs 180. In some implementations, the balance engine 135 applies a marginal relevance algorithm to generate the set of final listings. In some implementations, the balance engine 135 performs a relevance-diversity balance on the ranked listing pairs 180 to generate a set of final listings 185. In an example, the ranked listing pairs 180 de-duplicated and then only the item listings above a threshold diversity value are provided to the user device. The set of final listings 185 is transmitted to the user device 150. In an example, the balance engine 135 truncates the ranked listing of pairs 180 at some number of listings to generate the final listings 185. In an example, the balance engine 135 generates the final listings 185 from the ranked listing of pairs 180 by truncating the ranked listing of pairs 180 at some level of similarity.

[0030] FIGs. 3 through 5 illustrate an example set of item listings and how the method operates. In this example, the query 152 can be a phrase such as “pink 2024 tour t-shirt.” FIG. 3 illustrates an example of a full set of candidate item listings 170 determined to be relevant to this query using, for example, the retrieval engine 115 depicted in FIG. 1. In some implementations, the retrieval engine 115 uses an approximate nearest neighbor approach to produce the candidate item listings 170. The example set of candidate listings includes twelve item listings (300-1 to 300-12). This set of candidate item listings 170 may be returned after a search of the listing DB 140 is performed based on the query 152 received from the user device 150. The set of candidate item listings 170Attorney Docket No. 47166-0070W01includes, in this example, twelve listings for items: 300-1, 300-2, ... 300-12. It is apparent by inspection that items 300-1, 300-2, 300-3, and 300-4 are the same item — a T-shirt for a specific artist on a specific tour. Items 300-5 through 300-12 are T-shirts which are different in color or the design from the T-shirt of items 300-1 to 300-4.

[0031] The search optimization engine 120 determines a candidate item listing pairs 175 based on how similar each listing is to other item listings. In some implementations, the search optimization engine 120 uses an LLM 122 with, for example, a prompt. In some implementations, “Given this set of listings and associated attributes, identify the closest listing to every listing within the set according to how similar the products they represent look and feel.” In the example illustrated, the candidate item listing pairs 175 are as follows: (300-1, 300-2); (300-2, 300-1); (300-3, 300-1); (300-4, 300-1); (300-5, 300-1); (300-6, 300-8); (300-7, 300-9); (300-8, 300-6); (300-9, 300-7); (300-10, 300-11); (300-11, 300-12); (300-12, 300-9).

[0032] The ranking engine 130 ranks the candidate item listing pairs from most similar to least similar. In some implementations, the ranking engine 130 uses an LLM 122 with, for example, the prompt “Given this set of listing pairs, create an ordered list containing at the top pairs that are the most similar and at the bottom pairs that are the least similar - while listing ex aequo pairs, if any, on the same line.” The ranking engine 130 ranks each of the candidate item listing pairs based on how similar the two items of each pair appear. A conventional solution would have been to compare every ranked listing pair to all of the already-ranked listing pairs, which results in a computational cost that grows quadratically with the size of the set to be ranked. In contrast, an LLM can perform the ranking in one step, which results in a computational cost that only grows linearly with the size of the set to be ranked. In the example illustrated, because items 300-1 through 300-4 are identical, they have the same ranking. The example ranked listed pairs 180 are as follows:Most similar(300-1, 300-2); (300-2, 300-1); (300-3, 300-1); (300-4, 300-1)(300-5, 300-1)(300-7, 300-9); (300-9, 300-7); (300-11, 300-12); (300-12, 300-9)(300-6, 300-8); (300-8, 300-6)(300-10, 300-11)Least similarAttorney Docket No. 47166-0070W01

[0033] In some implementations, the similarity metric for pairing item listings may be based on a portion of the item listing, for example, only on the images provided. For example, a trained machine learning model can identify listings with similar images and similar associated image attributes (e.g., color, objects, features, background, etc.).

[0034] FIG. 4 illustrates a ranked list of item listings from the most similar to the least similar. The balance engine 135 removes duplicate pairs at the same level of similarity to produce a ranked list. In some implementations, the items are recursively removed based on the first item listing in the group of most similar pairs. In an example, after the first item of the most similar group is identified, any item paired with that first item is removed from the ranked list to avoid duplication. In some implementations, only items of the same similarity with the first item are removed from the ranked list, while others which are less similar than a first group are not excluded. In the example illustrated, the most similar pairs are (300-1, 300-2); (300-2, 300-1); (300-3, 300-1); and (300-4, 300-1). Items 300-2, 300-3, and 300-4 are identical to item 300-1 and providing all four to a user device would be duplicative. The first item listing 300-1 is provided as the first item listing of the ranked listing of pairs. The ranked list in the example starts with item 300-1 (the first item of the most similar grouping) and the other identical items (300-2, 300-3, and 300-4) are removed from the ranked listing. In addition, because the next most similar pair is item 300-5 also with item 300-1, item 300-5 is also removed from the ranked listing. (In some implementations, item 300-5 would remain in the ranked list rather than be removed, because the similarity between 300-1 and 300-5 is distinct from the similarity between 300-1 and 300-2, 300-3, and 300-4.) The next most similar grouping is items 300-7, 300-9, 300-11, and 300-12, so item 300-7 is the next item to populate the ranked list, and items 300-9, 300-11, and 300-12 are removed. Then items 300-6 and 300-8 are the next most similar, so item 300-6 is added to the ranked list and item 300-8 is removed. Lastly, items 300-10 and 300-11 are the least similar pairing, so item 300-10 is added to the ranked list and item 300-11 is removed from the ranked list. So, the resulting ranked list has removed duplicate (i.e., nearly similar) items, in addition to providing the items ranked based on their similarity with the most similar other item of the candidate item listings.

[0035] FIG. 5 illustrates a truncated ranked list as the final list returned to the user device as the optimized search results. The ranked list of item listings is truncated by the balance engine 135 based on a threshold diversity value. In an example, the truncated item is item 300-10 because itAttorney Docket No. 47166-0070W01satisfied (e.g., met or exceeded) the threshold diversity value. In some implementations, the threshold diversity value is a number of items which, if the interim listing exceeds this number, excludes all items beyond that number in the interim listing. For example, if the user device 150 submits with the query 152, a request that only the top 3 items be provided as part of the search optimization, then the balance engine 135 transmits only the top 3 items to the user device 150. In some implementations, the user device can also request the top N items, where N is any positive integer. In some implementations the balance engine 135 is configured to provide the top N listings, rather than receiving a threshold diversity value from the user device 150. In some implementations, the threshold diversity value is a threshold in a similarity between the first item on the ranked list and every other item on the ranked list. For example, the threshold diversity value can be a value of similarity which must be exceeded to be included in the final listings. For example, if similarities between the first item and the next four items on the ranked list are 0.98, 0.89, 0.74 and 0.43 and the threshold diversity value is a similarity of 0.5, then the last item is excluded from the final listings. In some implementations, the threshold diversity value is one of a number of item listings, a maximum difference in a similarity metric between two members of an item pair, or a threshold similarity metric. Other threshold diversity values can also be used. In the example illustrated in FIG. 5, the top three listings feature relevant offerings without duplication and also a reasonable range of diversity in the search results.

[0036] FIG. 6 illustrates a process flow for this method. The steps of the illustrated process can be performed using any suitable computing system. For purposes of illustration and ease of description, the process steps are described as being performed using the components of the system depicted in FIGs. 1-2. The process flow 600 includes receiving a search query (operation 610), performing a search to determine the item listings most relevant to the query (operation 620), generating the closest pairs among the relevant item listings (operation 630), ranking each pair of the set of closest pairs (operation 640), generating a set of final listings from the ranked set of closest pairs (operation 650), and presenting the truncated list of final listings in order of decreasing similarity (operation 660).

[0037] A device sends a search query to the exchange platform (operation 610). The exchange platform receives, from a user device, a query requesting identification of a type of item provided on the exchange platform. In some implementations, the user device sends a search query for anAttorney Docket No. 47166-0070W01item to the exchange platform. For example, the user device may submit the search query “pink 2024 tour t-shirt” to the exchange platform. In some implementations, the query submitted by the user device to the exchange platform can include a threshold diversity value to indicate how broad the search may vary from the rest of the query. In some implementations the threshold diversity value is received by the exchange platform from the user device. In some implementations, the threshold diversity value can be determined by the exchange platform or by a component of the exchange platform. In an example, the exchange platform can determine the threshold diversity value based on other data, for example, user device data, user metadata, account history, available item listings, and the like.

[0038] The exchange platform performs a search (operation 620). The exchange platform performs a search on item listings provided in a database of item listings to generate a plurality of candidate item listings responsive to the query. In some implementations the retrieval engine of the exchange platform performs an approximate nearest neighbor search between the query and an encoding of each of the item listings in the listing DB and returns a set of results (e.g., N results where N is an integer greater than 1). In some implementations, the retrieval engine of the exchange platform performs another type of search, for instance, a keyword search, and returns a list of the item listings with one or more attributes that include one or more keywords from the query.

[0039] The exchange platform generates closest pairs among the results of the search (operation 630). The exchange platform generates, using a trained LLM and based on the plurality of candidate item listings, a plurality of closest item listing pairs, wherein the generating includes, for each candidate item listing among the plurality of candidate item listings, identifying another candidate item listing that has attributes corresponding to attributes of the candidate item listing. In some implementations, the search optimization engine of the exchange platform uses the trained LLM to generate the most similar item listings (e.g., the closest pairs). In some implementations, the search optimization engine assigns to the first item listing, as its matched pair, a selected item listing with which the first item listing is the most similar and continues to the second item listing until each item listing has been paired with a different item listing with which it is the most similar. In some implementations, the search optimization engine calculates a similarity between the candidate item listing among the plurality of candidate item listings and another candidate itemAttorney Docket No. 47166-0070W01listing of the plurality of candidate item listings and pairing the candidate item listing with another candidate item listing with a greatest calculated similarity. Thus, the search optimization engine generates or extracts the pairs of item listings which are most similar to each other — the closest pairs. In an example, a prompt such as “Given this set of listings and associated attributes, identify the closest listing to every listing within the set according to how similar the products they represent look and feel” is submitted to the trained LLM. The trained LLM extracts from the set of listing the most similar pairs.

[0040] The ranking engine of the exchange platform ranks each pair of item listings (operation 640). The exchange platform ranks, using the trained LLM, the plurality of closest item listing pairs to obtain a ranked list of item pairs. In some implementations, the ranking engine, using the LLM, ranks the pairs of item listings generated in operation 630. In some implementations, a similarity is calculated between the paired items. In an example, if a first pair of item listings has a similarity of 0.99 and a second pair of item listings has a similarity of 0.5, then the first pair would be ranked ahead of the second pair. A similarity between item listings can be determined using, for example, a cosine similarity, a radial basis function (RBF) kernel, or an inverse Minkowski distance, but is not limited thereto. In some implementations, the trained LLM performs the ranking. In some implementations, the trained LLM is able to perform the ranking without resorting to precise similarity calculations and comparison. The trained LLM uses information distributed amongst the various nodes and parameters of the trained LLM. In some implementations, the ranking engine ranks the plurality of closest item listing pairs to obtain the ranked list of item pairs by ranking each item listing pair based on the calculated similarity between the item listing and the other item listing. The ranking engine of the exchange platform produces a ranked list of pairs of item listings, wherein the ranking is based on a degree of similarity calculated between the item listings of the pair. In an example, a prompt such as “Given this set of listing pairs, create an ordered list containing at the top pairs that are the most similar and at the bottom pairs that are the least similar - while listing ex aequo pairs, if any, on the same line” is submitted to the trained LLM. The trained LLM ranks each item from the item listing pairs based on the within-pair similarity.

[0041] The exchange platform generates a set of final listings (operation 650). The exchange platform generates, based on the ranked list of item pairs and the received query, a set of finalAttorney Docket No. 47166-0070W01listings. In some implementations, the balance engine of the exchange platform receives the ranked list of pairs of item listings from the ranking engine and generates the set of final listings. In some implementations, the balance engine, for each of the plurality of candidate item pairs, recursively removes any remaining pairs which include either item listing of the selected item pair. In some implementations, the balance engine eliminates duplicate item listings and truncates the list of items based on a threshold diversity value. In some implementations in eliminating duplicate item listings, the balance engine examines the ranked list of pairs of item listings and eliminates item listings which are identical (e.g., have the same, or very nearly the same, similarity). In the example shown in FIG. 3, the listings 300-1, 300-2, 300-3, and 300-4 were all identical and so 300-2 through 300-4 were eliminated from the set of final listings. In a conventional solution, in contrast, a balance engine can apply a maximal marginal relevance (MMR) algorithm to the ranked list of item pairs to generate the set of final listings. For example, the article “The use of MMR, diversitybased reranking for reordering documents and producing summaries” SIGIR ’98: Proceedings of the 21stannual international ACM SIGIR conference on research and development in information retrieval”, pp. 335-336 by Jaime Carbonell and Jade Goldstein provides an example of such an algorithm. In the article, a measurement of relevance and a measurement of diversity are presented and the linear combination is maximized to achieve a balance between relevance and diversity. The present method avoids the need to perform the calculations of the MMR algorithm and thus saves time and computing resources over that solution.

[0042] In addition to removing duplicate item listings, the balance engine truncates the list of item listing based on the threshold diversity value. In an example, the threshold diversity value is a positive integer, and the balance engine truncates the list of item listings at that number of item listings. In an example, the threshold diversity value is a similarity difference, and the balance engine truncates the list of item listings when the similarity between the pair of item listings exceeds the similarity value. For example, if the first pair of item listings has a similarity of 0.90 and the threshold diversity value is 0.10, then the balance engine truncates the final set of listings at any pair with a similarity below 0.80 = 0.90 - 0.10. In an example, the threshold diversity value is an absolute similarity value, and the balance engine truncates the list of item listings when the similarity between the pair of item listings falls below this value. For example, if the threshold diversity value is 0.76, then the balance engine truncates the list of item listings for all pairs withAttorney Docket No. 47166-0070W01a similarity at or below 0.76. In some implementations, the threshold diversity value is one of a number of item listings, a maximum difference in a similarity metric between two members of an item pair, or a threshold similarity metric. The threshold diversity value is not limited to the examples presented here. For example, a low threshold diversity value corresponds to excluding results which are not highly ranked in the ranked listing pairs and a high threshold diversity value corresponds to including results which are more lowly ranked in the ranked listing pairs. In some implementations, the threshold diversity value can be determined by the exchange platform or by a component of the exchange platform. In an example, the exchange platform can determine the threshold diversity value based on other data, for example, user device data, user metadata, account history, available item listings, and the like.

[0043] The exchange platform provides the final listings for display on the user device (operation 660). The exchange platform provides for display, to an application executing within the user device, the set of final listings that are responsive to the query. The set of final listings is the list of item listings which has been paired based on similarity with other item listings, ranked by similarity, de-duplicated, and truncated based on the threshold diversity value. This set of final listings is transmitted to the user device. The set of final listings thus represents a balance between a breadth of the original search query and a diversity of item listings presented to the user device. The candidate item listings can be many based on the match between the item listings in the listing DB and the search query. The final listings can be fewer since duplicate items have been removed from the set of final item listings. The threshold diversity value provides a limit on how broad the set of final item listings presented to the user device is. The user device can request a follow-on query with a larger threshold diversity value to receive more item listings of the set of final item listings. Alternatively, if there are too many item listings, a follow-on query may use a smaller threshold diversity value.

[0044] In this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.Attorney Docket No. 47166-0070W01

[0045] The subject matter and the actions and operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. The subject matter and the actions and operations described in this specification can be implemented as or in one or more computer programs, e.g., one or more modules of computer program instructions, encoded on a computer program carrier, for execution by, or to control the operation of, data processing apparatus. The carrier can be a tangible non-transitory computer storage medium. Alternatively or in addition, the carrier can be an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be or be part of a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. A computer storage medium is not a propagated signal.

[0046] FIG. 7 is a block diagram of computing devices 700, 750 that may be used to implement the systems and methods described in this specification, as either a client or as a server or plurality of servers. Computing device 700 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device 750 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, smartwatches, head-worn devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations described and / or claimed in this specification.

[0047] Computing device 700 includes a processor 702, memory 704, a storage device 706, a highspeed interface 708 connecting to memory 704 and high-speed expansion ports 710, and a low-speed interface 712 connecting to low-speed bus 714 and storage device 706. Each of the components 702, 704, 706, 708, 710, and 712, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor 702 can process instructions for execution within the computing device 700, including instructions stored in the memory 704 or on the storage device 706 to display graphical information for a GUI on anAttorney Docket No. 47166-0070W01external input / output device, such as display 716 coupled to high-speed interface 708. In other implementations, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices 700 may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi -processor system).

[0048] The memory 704 stores information within the computing device 700. In one implementation, the memory 704 is a computer-readable medium. In one implementation, the memory 704 is a volatile memory unit or units. In another implementation, the memory 704 is a non-volatile memory unit or units.

[0049] The storage device 706 is capable of providing mass storage for the computing device 700. In one implementation, the storage device 706 is a computer-readable medium. In various different implementations, the storage device 706 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 704, the storage device 706, or memory on processor 702.

[0050] The high-speed controller 708 manages bandwidth-intensive operations for the computing device 700, while the low-speed controller 712 manages lower bandwidth-intensive operations. Such allocation of duties is exemplary only. In one implementation, the high-speed controller 708 is coupled to memory 704, display 716 (e.g., through a graphics processor or accelerator), and to high-speed expansion ports 710, which may accept various expansion cards (not shown). In the implementation, low-speed controller 712 is coupled to storage device 706 and low-speed expansion port 714. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.Attorney Docket No. 47166-0070W01

[0051] The computing device 700 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server 720, or multiple times in a group of such servers. It may also be implemented as part of a rack server system 724. In addition, it may be implemented in a personal computer such as a laptop computer 722. Alternatively, components from computing device 700 may be combined with other components in a mobile device (not shown), such as device 750. Each of such devices may contain one or more of computing device 700, 750, and an entire system may be made up of multiple computing devices 700, 750 communicating with each other.

[0052] Computing device 750 includes a processor 752, memory 764, an input / output device such as a display 754, a communication interface 766, and a transceiver 768, among other components. The device 750 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components 750, 752, 764, 754, 766, and 768, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

[0053] The processor 752 can process instructions for execution within the computing device 750, including instructions stored in the memory 764. The processor may also include separate analog and digital processors. The processor may provide, for example, for coordination of the other components of the device 750, such as control of user interfaces, applications run by device 750, and wireless communication by device 750.

[0054] Processor 752 may communicate with a user through control interface 758 and display interface 756 coupled to a display 754. The display 754 may be, for example, a TFT LCD display or an OLED display, or other appropriate display technology. The display interface 756 may comprise appropriate circuitry for driving the display 754 to present graphical and other information to a user. The control interface 758 may receive commands from a user and convert them for submission to the processor 752. In addition, an external interface 762 may be provided in communication with processor 752, so as to enable near area communication of device 750 with other devices. External interface 762 may provide, for example, for wired communication (e.g., via a docking procedure) or for wireless communication (e.g., via Bluetooth or other such technologies).Attorney Docket No. 47166-0070W01

[0055] The memory 764 stores information within the computing device 750. In one implementation, the memory 764 is a computer-readable medium. In one implementation, the memory 764 is a volatile memory unit or units. In another implementation, the memory 764 is a non-volatile memory unit or units. Expansion memory 774 may also be provided and connected to device 750 through expansion interface 772, which may include, for example, a SIMM card interface. Such expansion memory 774 may provide extra storage space for device 750, or may also store applications or other information for device 750. Specifically, expansion memory 774 may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, expansion memory 774 may be provided as a security module for device 750, and may be programmed with instructions that permit secure use of device 750. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.

[0056] The memory may include for example, flash memory and / or MRAM memory, as discussed below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 764, expansion memory 774, or memory on processor 752.

[0057] Device 750 may communicate wirelessly through communication interface 766, which may include digital signal processing circuitry where necessary. Communication interface 766 may provide for communications under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication may occur, for example, through radio-frequency transceiver 768. In addition, short-range communication may occur, such as using a Bluetooth, WiFi, or other such transceiver (not shown). In addition, GPS receiver module 770 may provide additional wireless data to device 750, which may be used as appropriate by applications running on device 750.

[0058] Device 750 may also communicate audibly using audio codec 760, which may receive spoken information from a user and convert it to usable digital information. Audio codec 760 may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of deviceAttorney Docket No. 47166-0070W01750. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music fdes, etc.) and may also include sound generated by applications operating on device 750.

[0059] The computing device 750 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone 780, e.g., a smartphone. In some instances, the computing device 750 may be implemented as a tablet 782. Other types of the computing device 750 can include an extended reality device, e.g., an augmented reality device or a virtual reality device, a personal digital assistant, or another similar mobile device.

[0060] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0061] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms “machine-readable medium” “computer-readable medium” refers to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0062] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of what is being claimed, which is defined by the claims themselves, but rather as descriptions of features that can be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context ofAttorney Docket No. 47166-0070W01separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features can be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claim can be directed to a subcombination or variation of a subcombination.

[0063] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this by itself should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0064] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing can be advantageous.

Claims

Attorney Docket No. 47166-0070W01CLAIMS1. A computer-implemented method comprising:receiving, from a user device and by an exchange platform, a query requesting identification of a type of item provided on the exchange platform;performing a search on item listings provided in a database of item listings to generate a plurality of candidate item listings responsive to the query;generating, using a trained large language model (LLM) and based on the plurality of candidate item listings, a plurality of most similar item listing pairs, wherein the generating includes, for each candidate item listing among the plurality of candidate item listings,identifying another candidate item listing that has attributes corresponding to attributes of the candidate item listing;ranking, using the trained LLM, the plurality of most similar item listing pairs to obtain a ranked list of item pairs;generating, based on the ranked list of item pairs and the received query, a set of final listings; andproviding for display, to an application executing within the user device, the set of final listings that are responsive to the query.

2. The computer-implemented method of claim 1, wherein performing the search on item listings comprises performing an approximate nearest neighbor search on item listings provided in the database of item listings responsive to the query.

3. The computer-implemented method of claim 1, wherein identifying another candidate item listing that has attributes corresponding to attributes of the candidate item listing comprises, calculating a similarity between the candidate item listing among the plurality of candidate item listings and another candidate item listing of the plurality of candidate item listings and pairing the candidate item listing with another candidate item listing with a greatest calculated similarity.Attorney Docket No. 47166-0070W014. The computer-implemented method of any of claims 1-3, wherein ranking the plurality of most similar item listing pairs to obtain the ranked list of item pairs comprises ranking each item listing pair based on the calculated similarity between the item listing and the other item listing.

5. The computer-implemented method of any of claims 1-4, wherein generating the set of final listings comprises truncating the set of final listings based on a threshold diversity value.

6. The computer-implemented method of any of claims 1-5, wherein the threshold diversity value is received from the user device.

7. The computer-implemented method of any of claims 1-5, wherein the threshold diversity value is determined by the exchange platform.

8. The computer-implemented method of any of claims 1-7, wherein the threshold diversity value is one of a number of item listings, a maximum difference in a similarity metric between two members of an item pair, or a threshold similarity metric.

9. The computer-implemented method of any of claims 1-8, wherein generating the set of final listings comprises, for each of the ranked list of item pairs, recursively removing any remaining pairs which include either item listing of a selected item pair.

10. A system for performing the computer-implemented method of any of claims 1-9.

Images