The invention discloses a
historical record backtracking method based on LLM, and relates to the field of
data retrieval and
natural language processing, and the method comprises the steps: collecting various operation data of a user in a local computer, extracting text information in pictures and voices in combination with an OCR technology and an ASR technology, and storing the text information and related information in a
database in a unified format; the method comprises the following steps of:
processing collected multi-
modal data information, firstly performing de-duplication and cleaning on the collected data, then converting a collected screenshot into a video, and finally performing vector embedding on a
video image frame, an OCR (
Optical Character Recognition) result and a voice transcription text and storing the
video image frame, the OCR result and the voice transcription text into a vector
database; optimizing the pre-trained LLM in a specific direction by adopting a
fine tuning technology; according to the method, conversation with a historical timeline is realized,
user input is captured and stored in a query
queue, personalized response is generated through LLM interaction in combination with a
memory module and an RAG technology, and answers and related image or audio links are displayed.