Neural Network Content Relevance Scoring for Real-Time Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing resource matching systems face scalability issues and latency problems due to the complexity of evaluating and ranking potential matches in real-time, often resulting in inaccurate predictions and inefficient processing in transactional environments.
Innovation Solution
The use of an artificial neural network to score the relevance of content items for targets, allowing for precomputation and preselection of matches, which reduces data and processing needs in live environments by shifting demand to bulk processing environments, thereby accelerating transactional production.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If exhaustive exploration of potential matches is performed using MapReduce, then match completeness is improved, but processing speed and resource efficiency deteriorate
Solution Approach 1:
The system performs precomputation of relevance scores in a bulk processing environment before the live production system needs to make matches. This preliminary action shifts the computationally intensive work to an offline phase, allowing the online system to quickly retrieve precomputed scores without performing exhaustive exploration in real-time.
Solution Approach 2:
The matching process is divided into two separate phases: an offline bulk processing phase that performs exhaustive exploration and precomputation, and an online transactional phase that retrieves and uses precomputed scores. This segmentation allows each phase to be optimized independently for its specific requirements.
2Measurement precision
If complex analytics processing is used to improve accuracy of suitability prediction, then match accuracy is improved, but latency increases
Solution Approach 1:
Complex analytics processing is performed in advance during the offline phase to compute relevance scores with high accuracy. The results are stored and reused in the online phase, eliminating the need to repeat complex computations for each transaction and thus reducing latency while maintaining accuracy.
3Adaptability or versatility
If the system inventories thousands of resources for thousands of entities, then match coverage is improved, but data processing complexity increases
Solution Approach 1:
The system precomputes relevance scores for all resource-entity pairs in an offline bulk processing phase, transforming the complex real-time matching problem into a simpler retrieval and ranking task. This allows the system to handle large inventories without increasing online processing complexity.
Solution Approach 2:
Instead of performing complex matching computations for each query, the system creates precomputed copies of match relevance scores that can be quickly retrieved and used. This copying approach replaces complex real-time computation with simple data retrieval operations.
Data Source
AI summary
Herein are techniques to use an artificial neural network to score the relevance of content items for a target and techniques to rank the content items based on their scores. In embodiments, a computer uses a plurality of expansion techniques to identify expanded targets for a content item. For each of the expanded targets, the computer provides inputs to an artificial neural network to generate a relevance score that indicates a relative suitability of the content item for that target. The computer ranks the expanded targets based on the relevance score generated for each of the expanded targets. Based on the ranking, the computer selects a subset of targets from the available expanded targets as the expanded targets for whom the content item is potentially most relevant. The computer stores an association between the content item and each target in the subset of expanded targets.