Screenshot device tracing

Abstract

Disclosed in the present description are a screenshot device tracing method and apparatus, a storage medium, and an electronic device. The method comprises: acquiring a target screenshot in a transaction file processing scenario, and performing image encoding on the target screenshot to obtain image semantic representation data; then, acquiring image metadata of the target screenshot, and performing device model screening processing on the basis of the image metadata to obtain at least one piece of candidate device model data; and finally, on the basis of the candidate device model data, the image metadata, and the image semantic representation data, using a device identification large model to perform device model identification to obtain source device model information corresponding to the target screenshot.

Description

Screenshot device traceability Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method, apparatus, storage medium, and electronic device for tracing the source of a screenshot device. Background Technology

[0002] In the digital age, screenshots, as a common method of information capture, are widely used in various scenarios, such as business transactions, social media, online learning, and technical support. However, with the increased convenience of information sharing, the authenticity and source of screenshots have gradually attracted attention. Especially in business transaction scenarios, where screenshots are often used to collect user information, determining the source device of the screenshot to ascertain its authenticity is crucial for collecting user information. Summary of the Invention

[0003] This specification provides a method, apparatus, storage medium, and electronic device for tracing the source of screenshot devices, and the technical solution is as follows.

[0004] Firstly, this specification provides a method for tracing the source of a screenshot device. The method includes: acquiring a target screenshot in a transaction file processing scenario; performing image encoding processing on the target screenshot to obtain image semantic representation data; acquiring image metadata of the target screenshot; performing device model filtering processing based on the image metadata to obtain at least one candidate device model data; and performing device model detection processing using a device detection big data model based on the candidate device model data, the image metadata, and the image semantic representation data to obtain the source device model information corresponding to the target screenshot.

[0005] In one feasible implementation, the step of performing device model detection processing based on the candidate device model data, the image metadata, and the image semantic representation data using a device detection big data model to obtain the source device model information corresponding to the target screenshot includes: generating device detection prompt words based on the candidate device model data, the image metadata, and the image semantic representation data, and inputting the device detection prompt words into the device detection big data model for device model detection processing to obtain the source device model information corresponding to the target screenshot.

[0006] In one feasible implementation, the step of inputting the device detection prompt word into a device detection big model for device model detection processing to obtain the source device model information corresponding to the target screenshot includes: inputting the device detection prompt word into the device detection big model; using the device detection big model to perform feature fusion processing based on the image metadata and the image semantic representation data to obtain fused device feature data; and using the device detection big model to perform model matching processing based on the fused device feature data and the candidate device model data to obtain the source device model information corresponding to the target screenshot.

[0007] In one feasible implementation, the step of obtaining fused device feature data by performing feature fusion processing based on the image metadata and the image semantic representation data through the device detection big model includes: obtaining device display feature data by performing display feature extraction processing based on the image semantic representation data through the device detection big model; and obtaining fused device feature data by performing feature splicing processing based on the image metadata and the device display feature data through the device detection big model.

[0008] In one feasible implementation, the step of performing device model matching processing based on the fused device feature data and the candidate device model data using the device detection big model to obtain the source device model information corresponding to the target screenshot includes: determining the model feature data of at least one candidate device model based on the candidate device model data using the device detection big model; determining the matching degree between the model feature data and the fused device feature data using the device detection big model; determining the target device model feature data based on the matching degree; and generating the source device model information corresponding to the target screenshot based on the candidate device model corresponding to the target device model feature data.

[0009] In one feasible implementation, the step of performing device model filtering based on the image metadata to obtain at least one candidate device model data includes: performing field extraction processing on the image metadata to obtain target field data; obtaining a preset device database; and performing device model filtering based on the device database using the target field data to obtain at least one candidate device model data.

[0010] In one feasible implementation, the step of performing model filtering based on the target field data in the device database to obtain at least one candidate device model data includes: determining the target device model information indicated by the target field data; determining multiple reference device model information in the device database; calculating the model similarity between the reference device model information and the target device model information; and performing model filtering based on the model similarity using a similarity threshold to obtain at least one candidate device model data.

[0011] In one feasible implementation, the step of performing image encoding processing on the target screenshot to obtain image semantic representation data includes: performing privacy data hiding processing on the target screenshot to obtain a privacy-deprived screenshot, and performing image encoding processing on the privacy-deprived image to obtain image semantic representation data.

[0012] Secondly, this specification provides a screenshot device tracing apparatus, the apparatus comprising: an image processing module for acquiring a target screenshot in a transaction file processing scenario, and performing image encoding processing on the target screenshot to obtain image semantic representation data; a filtering processing module for acquiring image metadata of the target screenshot, and performing device model filtering processing based on the image metadata to obtain at least one candidate device model data; and a detection processing module for performing device model detection processing based on the candidate device model data, the image metadata, and the image semantic representation data using a device detection big data model to obtain the source device model information corresponding to the target screenshot.

[0013] Thirdly, this specification provides a computer storage medium having multiple instructions adapted for loading by a processor and executing the methods described above.

[0014] Fourthly, this specification provides a computer program product that stores at least one instruction, which is loaded by a processor and executes the method described above.

[0015] Fifthly, this specification provides an electronic device that may include: a memory and a processor; wherein the memory stores a computer program adapted to be loaded by the memory and to execute the methods described above.

[0016] The beneficial effects of the technical solutions provided in this specification include at least the following: The screenshot device tracing method provided in the embodiments of this specification, after acquiring a target screenshot in a transaction file processing scenario, performs image encoding processing on the target screenshot to obtain image semantic representation data. Then, it acquires the image metadata of the target screenshot, performs device model screening processing based on the image metadata to obtain at least one candidate device model data. Subsequently, based on the candidate device model data, image metadata, and image semantic representation data, it uses a large-scale device detection model to perform device model detection processing to obtain the source device model information corresponding to the target screenshot. Therefore, by inputting multi-faceted image data of the target screenshot and the initially screened candidate device model data into the large-scale device detection model, and utilizing the large-scale device detection model's ability to process multimodal data, the model information of the source device of the target screenshot can be accurately identified. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 is a schematic diagram of an application scenario of a screenshot device tracing method provided in the embodiments of this specification.

[0019] Figure 2 is a flowchart illustrating a screenshot device tracing method provided in an embodiment of this specification.

[0020] Figure 3 is a flowchart illustrating another method for tracing the source of a screenshot device provided in an embodiment of this specification.

[0021] Figure 4 is a schematic diagram of the structure of a screenshot device tracing device provided in an embodiment of this specification.

[0022] Figure 5 is a schematic diagram of the structure of a detection and processing unit provided in an embodiment of this specification.

[0023] Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. Detailed Implementation

[0024] To make the inventive objectives, features, and advantages of the embodiments in this specification more apparent and understandable, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments in this specification, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort are within the scope of protection of this specification.

[0025] In the description of this specification, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this specification, it should be noted that, unless otherwise expressly specified and limited, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. Those skilled in the art can understand the specific meaning of the above terms in this specification based on the specific circumstances. Furthermore, in the description of this specification, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0026] In related technologies, screenshots are a convenient way to collect user information. However, due to the simplicity of screenshot technology, the cost of forging and modifying screenshots is low, and traditional tampering detection techniques are difficult to identify, making it difficult to determine the authenticity of screenshots. In device traceability technologies, determining the type of device from which the screenshot originated and then comparing it with the user's actual device is an effective method to verify the authenticity of user screenshots. However, accurately detecting the source device of a screenshot is a technical problem that urgently needs to be solved.

[0027] To address the aforementioned technical problems, this specification will be described below with reference to specific embodiments.

[0028] Please refer to Figure 1, which is a schematic diagram of an application scenario for a screenshot device tracing method provided in an embodiment of this specification. As shown in Figure 1, the application scenario for screenshot device tracing may include at least a terminal cluster and a server 100.

[0029] In some embodiments, the terminal cluster may include at least one terminal, as shown in Figure 1, specifically including terminal 1 corresponding to user 1, terminal 2 corresponding to user 2, ..., terminal n corresponding to user n, where n is an integer greater than 0.

[0030] Each terminal in a terminal cluster can be a smart device with communication capabilities, including but not limited to: wearable devices, handheld devices, personal computers, tablets, smartphones, computing devices, or other processing devices connected to a wireless modem. In different networks, smart devices may be called by different names, such as: user equipment, access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, cellular phone, cordless phone, personal digital assistant (PDA), electronic devices in 5G networks or future evolved networks, etc.

[0031] Server 100 can be a standalone server device, such as a rack-mount, blade, tower, or cabinet-type server device, or a workstation, mainframe, or other hardware device with strong computing power; it can also be a server cluster composed of multiple servers, wherein the servers in the service cluster can be composed in a symmetrical manner, wherein each server is functionally and hierarchically equivalent in the transaction link, and each server can provide services to the outside world independently, wherein providing services independently can be understood as not requiring the assistance of other servers.

[0032] In some embodiments, the electronic device executing the screenshot device tracing method can be a server 100. The server 100 can establish a communication connection with at least one terminal in the terminal cluster, and the data interaction is completed during the screenshot device tracing process based on this communication connection. Specifically, the server 100 can be a backend server in a transaction file processing platform corresponding to the transaction file processing scenario. The transaction file processing platform not only includes a backend server but also has a frontend interface. At least one terminal in the terminal cluster establishes a communication connection with the server through the frontend interface to interact with data. For example, at least one terminal in the terminal cluster can upload a target screenshot in the transaction file processing platform through the frontend interface. Thus, the server obtains the target screenshot in the transaction file processing scenario, performs image encoding processing on the target screenshot to obtain image semantic representation data; obtains the image metadata of the target screenshot, performs device model screening processing based on the image metadata to obtain at least one candidate device model data; and performs device model detection processing using a device detection big model based on the candidate device model data, the image metadata, and the image semantic representation data to obtain the source device model information corresponding to the target screenshot.

[0033] It should be noted that server 100 establishes a communication connection with at least one terminal in the terminal cluster for interactive communication via a network. This network can be a wireless network or a wired network. Wireless networks include, but are not limited to, cellular networks, wireless LANs, infrared networks, or Bluetooth networks. Wired networks include, but are not limited to, Ethernet, universal serial bus (USB), or controller area networks. In one or more embodiments of the specification, technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network (such as target compressed packets). Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0034] In one embodiment, as shown in Figure 2, a method for tracing the origin of screenshot devices is proposed. This method can be implemented using a computer program and can run on a screenshot device tracing device based on the von Neumann architecture. The computer program can be integrated into an application or run as a standalone utility application.

[0035] Specifically, the methods for tracing the source of the screenshot device include the following.

[0036] S202, Obtain the target screenshot in the transaction file processing scenario, and perform image encoding processing on the target screenshot to obtain image semantic representation data.

[0037] In this context, the "transaction file processing scenario" refers to the process of tracing the device origin of images containing transaction materials. Device origination can be understood as the process of detecting the type of smart device that captured the image. The image can be captured by a camera installed on the acquisition device, or it can be a screenshot of the acquisition device's screen. Optionally, the transaction materials contained in the image can be user credentials of various transaction types. For example, in financial transactions, the transaction materials contained in the image could include a user's personal salary verification materials, such as bank statements, individual income tax records, etc.

[0038] The target screenshot refers to a screenshot that contains transactional material.

[0039] Image semantic representation data refers to the feature data obtained by encoding a target screenshot to represent the semantics of the image.

[0040] In some embodiments, a target screenshot uploaded by the user terminal to the transaction file processing platform can be obtained; the number of target screenshots can be one or more. Alternatively, a target screenshot can be obtained from a preset database corresponding to the transaction file processing platform. It is understood that the target screenshot is also a screenshot awaiting detection of the source device. For a description of the transaction file processing platform, please refer to the description of the relevant parts of the embodiment shown in Figure 1, which will not be detailed here.

[0041] Furthermore, image encoding tools can be used to perform image encoding processing on the target screenshot to obtain image semantic representation data.

[0042] S204, obtain the image metadata of the target screenshot, and perform device model filtering based on the image metadata to obtain at least one candidate device model data.

[0043] Image metadata refers to data used to describe the attribute information of an image. The data in image metadata may include, but is not limited to, the operating system of the device that captured the target screenshot, the compression algorithm corresponding to the target screenshot, the resolution of the target screenshot, and the image size of the target screenshot.

[0044] Candidate device model data refers to data used to describe device attribute information such as manufacturer, operating system, device type number, and resolution of various smart devices.

[0045] In some embodiments, an image file of the target screenshot can be obtained, and library functions can be used to read the image metadata of the target screenshot from the image file. Further, field data characterizing the operating system can be extracted from the image metadata, and this field data can be used to filter at least one candidate device model data in a preset device database; alternatively, field data characterizing the operating system and manufacturer can be extracted from the image metadata, and this field data can be used to filter at least one candidate device model data in the preset device database. The preset device database is used to store device attribute information such as the manufacturer, operating system, device type number, and resolution of different smart devices.

[0046] S206. Based on candidate device model data, image metadata, and image semantic representation data, a large device detection model is used to perform device model detection processing to obtain the source device model information corresponding to the target screenshot.

[0047] Among them, the source device model information is used to describe the specific model information and manufacturer of the smart device from which the target screenshot was collected.

[0048] In some embodiments, device detection prompts are generated based on candidate device model data, image metadata, and image semantic representation data. The device detection prompts are then input into a large device detection model for device model detection processing to obtain the source device model information corresponding to the target screenshot.

[0049] In the screenshot device tracing method provided in the embodiments of this specification, after acquiring a target screenshot in a transaction file processing scenario, image encoding processing is performed on the target screenshot to obtain image semantic representation data. Then, image metadata of the target screenshot is acquired, and device model filtering processing is performed based on the image metadata to obtain at least one candidate device model data. Subsequently, based on the candidate device model data, image metadata, and image semantic representation data, a large-scale device detection model is used to perform device model detection processing to obtain the source device model information corresponding to the target screenshot. Therefore, by inputting multi-faceted image data of the target screenshot and the initially filtered candidate device model data into the large-scale device detection model, and utilizing the large-scale device detection model's multimodal data processing capabilities, the model information of the source device of the target screenshot can be accurately identified.

[0050] Please refer to Figure 3, which is a flowchart illustrating another embodiment of a screenshot device tracing method provided in this specification. As shown in Figure 3, the method described in this specification embodiment may include the following steps.

[0051] S302, Obtain the target screenshot in the transaction file processing scenario, and perform image encoding processing on the target screenshot to obtain image semantic representation data.

[0052] In some embodiments, the specific method for obtaining the target screenshot can be found in the description of the relevant parts in the embodiment shown in Figure 2, and will not be detailed here. The step of performing image encoding processing on the target screenshot to obtain image semantic representation data may specifically include: performing privacy data hiding processing on the target screenshot to obtain a privacy-de-privacy screenshot, and performing image encoding processing on the privacy-de-privacy image to obtain image semantic representation data. It is understood that the target screenshot may include personal privacy information such as name and identity information. Privacy data in the target screenshot can be obscured or hidden through deblurring or other processes to obtain a privacy-de-privacy screenshot. Then, an image encoding tool is used to perform image encoding processing on the privacy-de-privacy image to obtain image semantic representation data. Image semantic representation data may specifically include, but is not limited to, text content feature data, top image pattern feature data, bottom image pattern feature data, software interface style feature data, etc. Top image pattern feature data may include, but is not limited to, feature data of patterns such as signal patterns, battery patterns, and time patterns. Bottom image pattern feature data may include feature data of patterns such as slider patterns. Software interface style feature data may include feature data of features such as software icons and software layouts.

[0053] S304: Obtain the image metadata of the target screenshot, and perform field extraction processing on the image metadata to obtain the target field data.

[0054] In some implementations, an image file of the target screenshot can be obtained, and library functions can be used to read the image metadata of the target screenshot from the image file. The image metadata can specifically include attribute information such as the operating system of the device capturing the target screenshot, the compression algorithm corresponding to the target screenshot, the resolution of the target screenshot, and the image size of the target screenshot. Further, specified fields and their values ​​are extracted from the image metadata to generate target field data including the specified fields and their values. The specified field extracted from the image metadata is either a first preset field or a second preset field. When the first preset field exists in the image metadata, the specified field is the first preset field; when the second preset field exists in the image metadata, the specified field is the second preset field. Specifically, the first preset field can be a field used to identify a first operating system, and the second preset field can be a field used to identify a second operating system. The first and second operating systems are different operating systems; for example, the first operating system can be Apple's iOS system, and the first preset field used to identify Apple's iOS system can be the userComment field; the second operating system can be Android, and the second preset field used to identify Android can be the Software field.

[0055] S306, Obtain a preset device database, and perform model filtering based on the target field data in the device database to obtain at least one candidate device model data.

[0056] The preset device database is used to store device attribute information such as manufacturers, operating systems, device type numbers, and resolutions of various smart devices.

[0057] Before implementing the screenshot device tracing method in the embodiments of this specification, a device database is constructed by collecting device attribute information of different smart devices. As new smart device products are released, the device database is continuously updated with the device attribute information of new smart devices.

[0058] In some embodiments, the acquired preset device database is the most recently updated device database, which is a rich and accurate device database. Further, performing a model filtering process based on the device database using target field data to obtain at least one candidate device model data may specifically include: determining the target device model information indicated by the target field data; determining multiple reference device model information in the device database; calculating the model similarity between the reference device model information and the target device model information; and performing model filtering based on the model similarity using a similarity threshold to obtain at least one candidate device model data.

[0059] Understandably, since the target field data includes a specified field and the values ​​of that specified field, the specified field is used to identify the operating system. The operating system of the device capturing the target screenshot can be determined by the value of the specified field, or the manufacturer and operating system of the device capturing the target screenshot can be determined by the value of the specified field. When the operating system of the device capturing the target screenshot can be determined by the value of the specified field, that operating system is used as the target device model information. At this time, the reference device model information can include the operating systems of different smart devices in the device database. The first model similarity between each reference device model information and the target device model information is calculated. The first model similarities greater than a similarity threshold can be arranged in descending order to obtain a first similarity sequence. A preset number of first model similarities in the first similarity sequence are used as the first similarity. The first device model data of the smart device corresponding to the first similarity is retrieved from the device database. This first device model data is used as candidate device model data, thus obtaining at least one candidate device model data. When the manufacturer and operating system of the target screenshot acquisition device can be determined by the value of a specified field, the manufacturer and operating system are used as the target device model information. At this time, each reference device model information can include the operating system and manufacturer of different smart devices stored in the device database. The second model similarity between each reference device model information and the target device model information is calculated. The second model similarities greater than the similarity threshold can be arranged in descending order to obtain a second similarity sequence. A preset number of second model similarities in the second similarity sequence are used as the second similarity. The second device model data of the smart device corresponding to the second similarity is obtained from the device database. The second device model data is used as candidate device model data to obtain at least one candidate device model data.

[0060] Therefore, by filtering a specified number of candidate device model data from the device database using the target field data, the system avoids inputting all device model data from the device database into the large model, thus reducing the amount of device model data input into the large model.

[0061] S308 generates device detection prompts based on candidate device model data, image metadata, and image semantic representation data. The device detection prompts are then input into the device detection big model for device model detection processing to obtain the source device model information corresponding to the target screenshot.

[0062] Among them, the device detection prompts are task description information used to instruct the large-scale device detection model to perform device model detection processing based on candidate device model data, image metadata, and image semantic representation data.

[0063] The device detection large model can be obtained by training a multimodal large model for the device tracing scenario in screenshots. The multimodal large model (MLLM) is a type of model that combines the natural language processing capabilities of the large language model (LLM) with the ability to understand and generate data from other modalities (such as visual and audio). It can process and understand multiple modalities of data. Unlike single-modal models, multimodal large models can integrate different types of data to provide richer and more accurate understanding and generation capabilities.

[0064] In some embodiments, a preset device detection prompt generation template can be obtained, and candidate device model data, image metadata, and image semantic representation data can be written into the device detection prompt generation template to obtain device detection prompt words.

[0065] Further, the step of inputting device detection prompts into the device detection big model for device model detection processing to obtain the source device model information corresponding to the target screenshot may include: A2: inputting device detection prompts into the device detection big model, and using the device detection big model to perform feature fusion processing based on image metadata and image semantic representation data to obtain fused device feature data; A4: using the device detection big model to perform model matching processing based on the fused device feature data and candidate device model data to obtain the source device model information corresponding to the target screenshot.

[0066] When performing step A2, the following steps can be performed: a2: Extract display features based on image semantic representation data using the device detection big model to obtain device display feature data; a4: Perform feature splicing based on image metadata and device display feature data using the device detection big model to obtain fused device feature data.

[0067] Among them, device display feature data refers to feature data used to characterize the user interface design of smart devices. In step a2, since image semantic feature data can include text content feature data, top image pattern feature data, bottom image pattern feature data, software interface style feature data, etc., the device detection big model can extract top image pattern feature data, bottom image pattern feature data, and software interface style feature data from the image semantic feature data to generate device display feature data that includes top image pattern feature data, bottom image pattern feature data, and software interface style feature data.

[0068] In step a4, the device detection big model can extract the operating system of the acquisition device of the target screenshot, the compression algorithm corresponding to the target screenshot, the resolution of the target screenshot, and the image size of the target screenshot from the image metadata. Based on the operating system of the acquisition device of the target screenshot, it generates operating system feature data; based on the compression algorithm corresponding to the target screenshot, it generates compression algorithm feature data; based on the resolution of the target screenshot, it generates resolution feature data; based on the image size of the target screenshot, it generates image size feature data. The device detection big model performs feature concatenation processing on the top pattern feature data, bottom pattern feature data, and software interface style feature data of the device display feature data, along with the operating system feature data, compression algorithm feature data, resolution feature data, and image size feature data, to obtain fused device feature data that includes all of the above feature data.

[0069] When performing step A4, the following steps can be performed: a6: Determine the model feature data of at least one candidate device model based on the candidate device model data through the device detection big model; a8: Determine the matching degree between the model feature data and the fused device feature data through the device detection big model, determine the target model feature data based on the matching degree, and generate the source device model information corresponding to the target screenshot based on the candidate device model corresponding to the target model feature data.

[0070] In step a6, the device detection big model extracts the top pattern feature data, bottom pattern feature data, software interface style feature data, operating system feature data, compression algorithm feature data, resolution feature data, and image size feature data of the candidate device model from the candidate device model data. The model feature data includes all of the above candidate feature data.

[0071] In step a8, for each device model feature data, the device detection large model generates top pattern feature pairs including candidate image top pattern feature data and image top pattern feature data; bottom pattern feature pairs including candidate image bottom pattern feature data and image bottom pattern feature data; interface style feature pairs including candidate software interface style feature data and software interface style feature data; operating system feature pairs including candidate operating system feature data and operating system feature data; compression algorithm feature pairs including candidate compression algorithm feature data and compression algorithm feature data; resolution feature pairs including candidate resolution feature data and resolution feature data; and size feature pairs including candidate image size feature data and image size feature data. Further, for each device model feature data, the device detection large model determines the top pattern similarity of the top pattern feature pairs, the bottom pattern similarity of the bottom pattern feature pairs, the interface style similarity of the interface style feature pairs, the operating system similarity of the operating system feature pairs, the compression algorithm similarity of the compression algorithm feature pairs, the resolution similarity of the resolution feature pairs, and the size similarity of the size feature pairs. Furthermore, for each device model's feature data, the device detection large model obtains the first weight corresponding to the top pattern similarity, the second weight corresponding to the bottom pattern similarity, the third weight corresponding to the interface style similarity, the fourth weight corresponding to the operating system similarity, the fifth weight corresponding to the compression algorithm similarity, the sixth weight corresponding to the resolution similarity, and the seventh weight corresponding to the size similarity. Further, for each device model's feature data, the device detection large model calculates the first product of the first weight and the top pattern similarity, the second product of the second weight and the bottom pattern similarity, the third product of the third weight and the interface style similarity, the fourth product of the fourth weight and the operating system similarity, the fifth product of the fifth weight and the compression algorithm similarity, the sixth product of the sixth weight and the resolution similarity, and the seventh product of the seventh weight and the size similarity. The sum of the first, second, third, fourth, fifth, sixth, and seventh products is then calculated to obtain the matching degree. Therefore, each model feature data corresponds to a matching degree. The matching degrees corresponding to the model feature data are sorted in descending order to obtain a matching degree sequence. The model feature data corresponding to the first matching degree in the matching degree sequence is taken as the target model feature data. Source device model information is generated, which includes the candidate device models corresponding to the target model feature data. This source device model information is the model information of the source device from which the target screenshot is collected. The source device model information may include the manufacturer and specific model of the smart device.

[0072] Optionally, the training process of the device detection large model in the embodiments of this specification can be as follows: An initial device detection large model for the screenshot device tracing scenario is created using a basic multimodal large model; sample screenshots are acquired, and image encoding processing is performed on the sample screenshots to obtain sample image semantic representation data; sample image metadata of the sample screenshots is acquired; sample candidate device model data corresponding to the sample screenshots is determined; sample data including sample candidate device model data, sample image metadata, and sample image semantic representation data is generated; source device model information tags are labeled on the sample data; the initial device detection large model is trained at least once using the sample data; during the model training process, the initial device detection large model is used to detect device models on the sample data to obtain predicted source device model information; the model parameters of the initial device detection large model are adjusted based on the predicted source device model information and the source device model information tags to obtain the trained device detection large model.

[0073] Specifically, the execution of creating an initial device detection model for the screenshot device tracing scenario using a basic multimodal large model can be as follows: obtaining the basic multimodal large model, creating a screenshot device tracing scenario adaptation module and a large language generative module based on the basic multimodal large model, and combining the large language generative module and the screenshot device tracing scenario adaptation module to form the initial device detection model.

[0074] Specifically, the model training process of the initial device detection large model can be as follows: Input sample data into the initial device detection large model for at least one round of model training to obtain predicted source device model information; calculate the model's comprehensive loss based on the predicted source device model information and the source device model information labels; adjust the model parameters of the screenshot device tracing scenario adaptation module in the initial device detection large model based on the comprehensive model loss, while keeping the model parameters of the large language generation module unchanged. This process continues until the model training termination condition is met, resulting in the large language generation module and the screenshot device tracing scenario adaptation module. Finally, the model fusion of the large language generation module and the screenshot device tracing scenario adaptation module is completed, yielding the trained device detection large model.

[0075] Specifically, the model fusion of the large language generation module and the screenshot device tracing scenario adaptation module can be as follows: the model structure layer weights of the screenshot device tracing scenario adaptation module and the large language generation module are fused. By determining the target model structure layer corresponding to the model structure layer weights in the large language generation module, the model structure layer parameters of the target model structure layer are fused with the model structure layer weights. The model structure layer weights of the screenshot device tracing scenario adaptation module may only partially correspond to and exist in all model structure layers in the basic large language model. By updating the parameters of the model structure layer based on the model structure layer weights for this part of the target model structure layer, the reference update process for the weights of all model structure layers is completed in this way, thereby obtaining the device detection large model.

[0076] Optionally, the model's training termination conditions may include, for example, the loss function value being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold. Specific training termination conditions can be determined based on actual circumstances and are not specifically limited here.

[0077] The screenshot device tracing method provided in this specification involves: acquiring a target screenshot in a transaction file processing scenario; performing image encoding on the target screenshot to obtain image semantic representation data; acquiring image metadata of the target screenshot; extracting target field data from the image metadata; acquiring a preset device database; performing device model filtering based on the target field data in the device database to obtain at least one candidate device model; generating device detection prompts based on the candidate device model data, image metadata, and image semantic representation data; and inputting the device detection prompts into a large device detection model for device model detection processing to obtain the source device model information corresponding to the target screenshot. Thus, by filtering a specified number of candidate device model data from the device database using the target field data, all device model data in the database are avoided from being input into the large model, reducing the amount of device model data input into the large model. By inputting the candidate device model data, image metadata, and image semantic representation data into the large device detection model, the data processing capabilities of the multimodal device detection model are utilized to accurately identify the source device model information corresponding to the target screenshot.

[0078] The screenshot device tracing device provided in the embodiments of this specification will now be described in detail with reference to Figure 4. It should be noted that the screenshot device tracing device shown in Figure 4 is used to execute the methods of the embodiments shown in Figures 1 to 3 of this specification. For ease of explanation, only the parts related to the embodiments of this specification are shown. For specific technical details not disclosed, please refer to the embodiments shown in Figures 1 to 3 of this specification.

[0079] Please refer to Figure 4, which shows a schematic diagram of the structure of the screenshot device tracing device according to an embodiment of this specification. This screenshot device tracing device 1 can be implemented as all or part of a device through software, hardware, or a combination of both. According to some embodiments, the screenshot device tracing device 1 includes an image processing module 11, a filtering processing module 12, and a detection processing module 13, specifically used for: the image processing module 11, for acquiring a target screenshot in a transaction file processing scenario, and performing image encoding processing on the target screenshot to obtain image semantic representation data; the filtering processing module 12, for acquiring image metadata of the target screenshot, and performing device model filtering processing based on the image metadata to obtain at least one candidate device model data; and the detection processing module 13, for performing device model detection processing based on the candidate device model data, the image metadata, and the image semantic representation data using a device detection big data model to obtain the source device model information corresponding to the target screenshot.

[0080] Optionally, the detection processing module 13 includes: a detection processing unit 131, used to generate device detection prompts based on the candidate device model data, the image metadata, and the image semantic representation data, and input the device detection prompts into the device detection big model to perform device model detection processing to obtain the source device model information corresponding to the target screenshot.

[0081] Optionally, please refer to the structural schematic diagram of the detection processing unit 131 shown in Figure 5. The detection processing unit 131 includes a feature fusion unit 1311 and a device model matching unit 1312. Specifically, the feature fusion unit 1311 is used to input the device detection prompt words into the device detection big model, and obtain fused device feature data by performing feature fusion processing based on the image metadata and the image semantic representation data through the device detection big model; the device model matching unit 1312 is used to perform device model matching processing based on the fused device feature data and the candidate device model data through the device detection big model, and obtain the source device model information corresponding to the target screenshot.

[0082] Optionally, the feature fusion unit 1311 is specifically used to: extract display features based on the image semantic representation data using the device detection big model to obtain device display feature data; and perform feature splicing processing based on the image metadata and the device display feature data using the device detection big model to obtain fused device feature data.

[0083] Optionally, the device matching unit 1312 is specifically used for: determining device feature data of at least one candidate device model based on the candidate device model data through the device detection big model; determining the matching degree between the device feature data and the fused device feature data through the device detection big model; determining target device feature data based on the matching degree; and generating source device model information corresponding to the target screenshot based on the candidate device model corresponding to the target device feature data.

[0084] Optionally, the filtering processing module 12 includes: a first filtering unit, used to perform field extraction processing on image metadata to obtain target field data; and a second filtering unit, used to obtain a preset device database, and perform model filtering processing based on the target field data using the device database to obtain at least one candidate device model data.

[0085] Optionally, the second filtering unit is specifically used for: determining the target device model information indicated by the target field data, determining multiple reference device model information in the device database; calculating the model similarity between the reference device model information and the target device model information, and performing model filtering processing based on the model similarity using a similarity threshold to obtain at least one candidate device model data.

[0086] Optionally, the image processing module 11 is specifically used to: perform privacy data hiding processing on the target screenshot to obtain a privacy-deprivation screenshot, and perform image encoding processing on the privacy-deprivation image to obtain image semantic representation data.

[0087] The screenshot device tracing device provided in this specification, after acquiring a target screenshot in a transaction file processing scenario, performs image encoding processing on the target screenshot to obtain image semantic representation data. Then, it acquires the image metadata of the target screenshot and performs device model filtering processing based on the image metadata to obtain at least one candidate device model data. Subsequently, based on the candidate device model data, image metadata, and image semantic representation data, it uses a device detection big data model to perform device model detection processing to obtain the source device model information corresponding to the target screenshot. Therefore, by inputting multi-faceted image data of the target screenshot and the initially filtered candidate device model data into the device detection big data model, and utilizing the device detection big data model's multimodal data processing capabilities, the device model information of the source device of the target screenshot can be accurately identified.

[0088] Please refer to Figure 6, which shows a schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this specification. The electronic device in this specification may include one or more of the following components: a processor 110, a memory 120, an input device 130, an output device 140, and a bus 150. The processor 110, memory 120, input device 130, and output device 140 may be connected to each other via the bus 150.

[0089] Processor 110 may include one or more processing cores. Processor 110 connects to various parts of the terminal using various interfaces and lines, and performs various functions and processes data of terminal 100 by running or executing instructions, programs, code sets, or instruction sets stored in memory 120, and by calling data stored in memory 120. Optionally, processor 110 may be implemented using at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). Processor 110 may integrate one or more of the following: central processing unit (CPU), graphics processing unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 110 and may be implemented separately using a communication chip.

[0090] The memory 120 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 120 may include non-transitory computer-readable storage medium. The memory 120 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (e.g., touch function, sound playback function, image playback function, etc.), instructions for implementing the various method embodiments described below, etc. The operating system may be the Android system, including systems deeply developed based on the Android system, the iOS system developed by Apple Inc., including systems deeply developed based on the iOS system, or other systems.

[0091] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to establish data communication between the third-party applications and the operating system. This would allow the operating system to obtain the current scenario information of the third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.

[0092] The input device 130 is used to receive input instructions or data, and includes, but is not limited to, a keyboard, mouse, camera, microphone, or touch device. The output device 140 is used to output instructions or data, and includes, but is not limited to, a display device and a speaker. In one example, the input device 130 and the output device 140 can be combined, and the input device 130 and the output device 140 can be a touch display screen.

[0093] The touch display screen can be designed as a full-screen, curved screen, or irregularly shaped screen. It can also be designed as a combination of a full-screen and a curved screen, or a combination of an irregularly shaped screen and a curved screen; however, this specification does not limit the specific design of the embodiments described herein.

[0094] In addition, those skilled in the art will understand that the structure of the electronic device shown in the above figures does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the electronic device may also include radio frequency circuits, input units, sensors, audio circuits, Wireless Fidelity (WiFi) modules, power supplies, Bluetooth modules, etc., which will not be described in detail here.

[0095] In the electronic device shown in Figure 6, the processor 110 can be used to call the program of the screenshot device tracing method stored in the memory 120, and specifically perform the following operations: obtain the target screenshot in the transaction file processing scenario, perform image encoding processing on the target screenshot to obtain image semantic representation data; obtain the image metadata of the target screenshot, perform device model screening processing based on the image metadata to obtain at least one candidate device model data; perform device model detection processing using a device detection big model based on the candidate device model data, the image metadata, and the image semantic representation data to obtain the source device model information corresponding to the target screenshot.

[0096] In one embodiment, when the processor 110 performs the step of performing device model detection processing based on the candidate device model data, the image metadata, and the image semantic representation data using a device detection big model to obtain the source device model information corresponding to the target screenshot, it specifically performs the following operations: generating device detection prompt words based on the candidate device model data, the image metadata, and the image semantic representation data, and inputting the device detection prompt words into the device detection big model to perform device model detection processing to obtain the source device model information corresponding to the target screenshot.

[0097] In one embodiment, when the processor 110 performs the step of inputting the device detection prompt into the device detection big model to perform device model detection processing to obtain the source device model information corresponding to the target screenshot, it specifically performs the following operations: inputting the device detection prompt into the device detection big model, and using the device detection big model to perform feature fusion processing based on the image metadata and the image semantic representation data to obtain fused device feature data; and using the device detection big model to perform model matching processing based on the fused device feature data and the candidate device model data to obtain the source device model information corresponding to the target screenshot.

[0098] In one embodiment, when the processor 110 performs the step of obtaining fused device feature data by performing feature fusion processing based on the image metadata and the image semantic representation data through the device detection big model, it specifically performs the following operations: obtaining device display feature data by performing display feature extraction processing based on the image semantic representation data through the device detection big model; and obtaining fused device feature data by performing feature splicing processing based on the image metadata and the device display feature data through the device detection big model.

[0099] In one embodiment, when the processor 110 performs the step of performing device model matching processing based on the fused device feature data and the candidate device model data through the device detection big model to obtain the source device model information corresponding to the target screenshot, it specifically performs the following operations: determining the model feature data of at least one candidate device model based on the candidate device model data through the device detection big model; determining the matching degree between the model feature data and the fused device feature data through the device detection big model; determining the target device model feature data based on the matching degree; and generating the source device model information corresponding to the target screenshot based on the candidate device model corresponding to the target device model feature data.

[0100] In one embodiment, when the processor 110 performs the step of filtering device models based on the image metadata to obtain at least one candidate device model data, it specifically performs the following operations: extracting fields from the image metadata to obtain target field data; obtaining a preset device database; and performing device model filtering based on the device database using the target field data to obtain at least one candidate device model data.

[0101] In one embodiment, when the processor 110 performs the step of performing model filtering based on the target field data in the device database to obtain at least one candidate device model data, it specifically performs the following operations: determining the target device model information indicated by the target field data, determining multiple reference device model information in the device database; calculating the model similarity between the reference device model information and the target device model information, and performing model filtering based on the model similarity using a similarity threshold to obtain at least one candidate device model data.

[0102] In one embodiment, when the processor 110 performs the step of image encoding processing on the target screenshot to obtain image semantic representation data, it specifically performs the following operations: performs privacy data hiding processing on the target screenshot to obtain a privacy-deprived screenshot, and performs image encoding processing on the privacy-deprived image to obtain image semantic representation data.

[0103] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this specification are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the target screenshots involved in this specification were all obtained under full authorization.

[0104] This specification also provides a computer-readable storage medium storing at least one instruction that is executed by a processor to implement the screenshot device tracing method as described in the above embodiments.

[0105] This specification also provides a computer program product that stores at least one instruction, which is loaded and executed by the processor to implement the screenshot device tracing method described in the above embodiments.

[0106] Those skilled in the art will recognize that the functions described in the embodiments of this specification in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0107] The above description is merely an optional embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification shall be included within the protection scope of this specification.

[0108] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

Claims

1. A method for tracing the origin of a screenshot device, the method comprising: Obtain a target screenshot in a transaction file processing scenario, and perform image encoding processing on the target screenshot to obtain image semantic representation data; Obtain the image metadata of the target screenshot, and perform device model filtering processing based on the image metadata to obtain at least one candidate device model data; Based on the candidate device model data, the image metadata, and the image semantic representation data, a large device detection model is used to perform device model detection processing to obtain the source device model information corresponding to the target screenshot.

2. The method according to claim 1, wherein the step of performing device model detection processing based on the candidate device model data, the image metadata, and the image semantic representation data using a large device detection model to obtain the source device model information corresponding to the target screenshot includes: Based on the candidate device model data, the image metadata, and the image semantic representation data, device detection prompts are generated. The device detection prompts are then input into the device detection big model for device model detection processing to obtain the source device model information corresponding to the target screenshot.

3. The method according to claim 2, wherein inputting the device detection prompt word into the device detection large model for device model detection processing to obtain the source device model information corresponding to the target screenshot includes: The device detection prompt words are input into the device detection model, and the device detection model performs feature fusion processing based on the image metadata and the image semantic representation data to obtain fused device feature data. The device detection big model performs model matching processing based on the fused device feature data and the candidate device model data to obtain the source device model information corresponding to the target screenshot.

4. The method according to claim 3, wherein obtaining fused device feature data by performing feature fusion processing based on the image metadata and the image semantic representation data using the device detection large model includes: The device display feature data is obtained by performing display feature extraction processing on the image semantic representation data using the device detection large model; The device detection big model performs feature stitching processing based on the image metadata and the device display feature data to obtain fused device feature data.

5. The method according to claim 3, wherein the step of performing model matching processing based on the fused device feature data and the candidate device model data through the device detection big model to obtain the source device model information corresponding to the target screenshot includes: Based on the candidate device model data, the device detection big model determines the model feature data of at least one candidate device model. The matching degree between the device model feature data and the fused device feature data is determined by the device detection big model. The target device model feature data is determined based on the matching degree. The source device model information corresponding to the target screenshot is generated based on the candidate device model corresponding to the target device model feature data.

6. The method according to claim 1, wherein the step of performing device model filtering based on the image metadata to obtain at least one candidate device model data includes: Extract the target field data by performing field extraction processing on the image metadata; Obtain a preset device database, and perform model filtering based on the target field data in the device database to obtain at least one candidate device model data.

7. The method according to claim 6, wherein the step of performing model filtering based on the target field data in the device database to obtain at least one candidate device model data includes: Determine the target device model information indicated by the target field data, and determine multiple reference device model information in the device database; Calculate the model similarity between the reference device model information and the target device model information, and perform model screening processing based on the model similarity using a similarity threshold to obtain at least one candidate device model data.

8. The method according to claim 1, wherein the image encoding processing of the target screenshot to obtain image semantic representation data includes: The target screenshot is subjected to privacy data hiding processing to obtain a privacy-deprivation screenshot, and the privacy-deprivation image is subjected to image encoding processing to obtain image semantic representation data.

9. A device for tracing the origin of a screenshot device, the device comprising: The image processing module is used to acquire target screenshots in a transaction file processing scenario, and to perform image encoding processing on the target screenshots to obtain image semantic representation data; The filtering processing module is used to obtain image metadata of the target screenshot, and perform device model filtering processing based on the image metadata to obtain at least one candidate device model data; The detection and processing module is used to perform device model detection processing based on the candidate device model data, the image metadata, and the image semantic representation data using a large device detection model, so as to obtain the source device model information corresponding to the target screenshot.

10. A computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the method as claimed in any one of claims 1 to 8.

11. A computer program product storing at least one instruction, said at least one instruction being loaded by a processor and executing the method as described in any one of claims 1 to 8.

12. An electronic device, comprising: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed as described in any one of claims 1 to 8.

Images