Active-Screener Recall Estimation for Bibliographic Screening
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The time-consuming and error-prone process of human screening in systematic reviews, where researchers must review thousands of citations to achieve 100% recall, lacks an efficient method to determine when all relevant documents have been identified, leading to unnecessary screening of irrelevant documents.
Innovation Solution
The Active-Screener web-based software tool employs state-of-the-art machine learning algorithms to estimate recall by sorting bibliographic references based on relevance, allowing human screeners to stop screening when a sufficient number of relevant documents are found, using a statistical model to predict the number of remaining relevant articles and providing real-time feedback.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If human screeners review all citations to achieve 100% recall, then all relevant documents are identified, but the time required becomes enormously long
Solution Approach 1:
The system performs preliminary sorting of citations by relevance before human screeners review them. This preliminary action ranks citations so that the most relevant ones appear first, allowing screeners to identify relevant documents more quickly without having to review every single citation to achieve high recall
Solution Approach 2:
The system provides real-time feedback to screeners about their screening progress and the characteristics of remaining unscreened citations. This feedback mechanism helps screeners understand how many relevant documents remain and adjust their screening strategy accordingly, reducing unnecessary review time while maintaining recall
2Loss of time
If citations are sorted by relevance, then screening time is reduced, but screeners cannot know when to stop screening without reviewing all documents
Solution Approach 1:
The system continuously provides feedback to screeners about the estimated number of relevant documents remaining in the unscreened set. This feedback is based on the sorted order and characteristics of already-screened citations, allowing screeners to make informed decisions about when to stop screening while maintaining high recall
Solution Approach 2:
The system allows screeners to perform partial screening by reviewing only a portion of the sorted citations rather than all of them. By providing estimates of remaining relevant documents, the system enables screeners to stop at an appropriate point without needing to complete full review, accepting slightly less than 100% recall in exchange for significant time savings
Data Source
AI summary
Methods and systems for estimating recall while screening an ordered list of bibliographic references are provided. According to one embodiment, a method includes: sorting a list of bibliographic references according to a sorting algorithm to produce a first list in order from most to least relevant; selecting, from the list, the most relevant reference, and displaying, to a human screener, information associated with the selected reference; and receiving the screener's judgment of the relevance of the selected reference. If sufficiently relevant, the selected reference is moved from the first list to a second list. The received indication is used to re-sort the remaining references in the first list. A statistical model is used to estimate the number of relevant references remaining. That estimate is displayed to the screener. The process ends when the screener determines, based on the displayed estimate, that a sufficient number of relevant references has been found.