Method and system for providing web navigation service on basis of action agent

Abstract

The present invention relates to a method and system for providing a web navigation service on the basis of an action agent , in which an AI model is trained with demonstration data in which actual web page manipulation is recorded, an action plan to be automatically performed by the trained AI is generated, and a web page is actually manipulated according to the generated plan to complete a requested task.

Description

Method and system for providing web navigation services based on action agents

[0001] The present invention relates to a method and system for providing a web navigation service based on an action agent. More specifically, it relates to a method and system for providing a web navigation service based on an action agent that learns user behavior patterns and physically operates a web or app directly.

[0002] Large Language Models (LLMs) are generative AI—that is, artificial intelligence—that generate appropriate answers by grasping context after repeated training with vast amounts of text, and are being utilized in various fields such as AI assistants, email composition, image generation, and coding.

[0003] On the other hand, while these Large-Scale Language Models (LLMs) focus on language processing and excel at language generation and comprehension, they have limitations in selecting the optimal action among multiple options or handling various interaction elements in complex environments. For example, they may be unsuitable for situations requiring actions such as "clicking a login button" or "entering an ID" in large-scale workspaces like websites.

[0004] To overcome the limitations of such Large Language Models (LLMs), Large Action Models (LAMs) were developed to directly operate websites and apps by learning user behavior patterns.

[0005] Large-scale Action Models (LAMs) are specialized models primarily for predicting and processing user actions in a task context; therefore, when a user performs various actions such as clicking, selecting, or dragging, they can process these actions in real time and predict the appropriate next action.

[0006] However, Large-Scale Behavior Models (LAMs) tend to rely heavily on behavioral patterns learned in a specific domain, so they may not function properly in new environments or domains where they have not been trained.

[0007] In addition, there is a possibility that all processes of the task may fail to be achieved because only information or interaction elements exposed on the main page (e.g., menus and / or buttons) are detected on the web page, and invisible or hidden information or interaction elements are not found.

[0008] Furthermore, demo data is collected to train small-scale models so that large-scale behavioral models (LAMs) provide accurate results. While the licensing conditions of the sources and sub-information included in each demo data set must be checked individually, it is virtually impossible for an individual to review the licenses for all data without error, as the volume of demo data is not only massive but also steadily increasing.

[0009] In other words, the need for the invention of a more advanced actionable agent is emerging to solve the aforementioned problem.

[0010] The present invention was devised to solve the problems of the prior art described above, and aims to provide a method and system for providing a web navigation service based on an action agent that infers preconditions for an assigned task based on an image-based intent extractor and a text-based guideline extractor.

[0011] In addition, the present invention aims to provide a method and system for providing a web navigation service based on an action agent that accumulates the inference process of demo data to train small language models (LMs) to be specialized for the domain.

[0012] In addition, the present invention aims to provide a method and system for providing a web navigation service based on an action agent that detects interaction elements that are not visible or are hidden on a web page and performs an action.

[0013] In addition, the present invention aims to provide a method and system for providing a web navigation service based on an action agent that automatically verifies license compliance for demo data.

[0014] However, the technical problems that the present invention and the embodiments of the present invention aim to solve are not limited to the technical problems described above, and other technical problems may exist.

[0015] A method for providing a web navigation service based on an action agent according to an embodiment of the present invention is a method in which a computing system including memory and a processor provides a web navigation service based on an action agent, comprising: a step of constructing a demo set database by generating a demo set including at least one action performed for a first intention on a first demo page; a step of performing machine learning training on at least one machine learning model based on the constructed demo set database; a step of obtaining a task for a first web page based on user input; a step of determining a first action according to an action determination process based on the trained machine learning models; a step of determining whether to perform the determined first action to generate a schedule in which the first action to the nth action is planned; and a step of providing a web navigation service that physically operates the first web page according to the generated schedule.

[0016] Additionally, the step of generating the demo set comprises: acquiring first log data including a plurality of actions performed to achieve a predetermined task on a first demo page; determining a first intention regarding the acquired first log data; and generating a demo set including the first demo page, the first log data, and the first intention, and storing it in a demo set database.

[0017] Additionally, the step of generating the demo set further includes the step of performing prompt engineering to extract and display interaction elements included in a first demo page for a first machine learning model, the step of matching a first demo image, which is a video recorded by tracing a cursor on the first demo page, to the demo set, and the step of performing machine learning training for the first machine learning model for which the prompt engineering was performed, based on the first demo image.

[0018] Additionally, the step of performing prompt engineering to extract and display the interaction elements includes parsing at least one of the HTML structure, buttons, input fields, and layout of the first web page based on a screenshot of the first web page, and visualizing at least one of the parsed interaction elements.

[0019] Additionally, the step of performing machine learning training based on the demo set database includes the step of training a first machine learning model to provide a plurality of intentions included in the demo set as output data using the demo set as input data, and the first machine learning model is trained to use a predetermined algorithm to sort the output plurality of intentions from a lower goal to a higher goal, and to generate and provide the schedule planned to sequentially perform the first action to the nth action to achieve the task through the sorted plurality of intentions.

[0020] Additionally, the step of performing machine learning training based on the demo set database includes the step of training a second machine learning model to provide a guideline, which is information representing the schedule in natural language text format, as output data using the demo set as input data, and the second machine learning model is trained to convert the schedule into a guideline using conjunctions and context as keywords.

[0021] Additionally, the step of acquiring a task for the first web page comprises providing a web navigation interface for a user to input a task in a natural language format, and performing a predetermined natural language processing on the acquired task based on the web navigation interface.

[0022] Additionally, the step of determining the first action according to the above action determination process includes the step of extracting interaction elements by analyzing the first web page based on the first machine learning model on which the prompt engineering was performed, and the step of determining detailed information including location information and descriptive information regarding the extracted interaction elements.

[0023] Additionally, the step of generating a schedule in which the first to nth actions are planned includes the step of extracting intentions and guidelines based on the first and second machine learning models for the acquired task, the step of generating a schedule according to the extracted intentions and guidelines by reflecting detailed information about the interaction elements, and the step of performing a first action on the first web page according to the generated schedule.

[0024] Meanwhile, a system for providing a web navigation service based on an action agent according to an embodiment of the present invention comprises: at least one memory; and at least one processor that reads at least one application stored in the memory and provides a web navigation service based on the action agent. The instructions of the processor include: creating a demo set including at least one action performed for a first intention on a first demo page to build a demo set database; performing machine learning training on at least one machine learning model based on the built demo set database; acquiring a task for a first web page based on user input; determining a first action according to an action determination process based on the trained machine learning models; determining whether to perform the determined first action to generate a schedule in which the first action to the nth action is planned; and providing a web navigation service that physically operates the first web page according to the generated schedule.

[0025] A method and system for providing web navigation services based on an action agent according to an embodiment of the present invention infers preconditions for an assigned task based on an image-based intent extractor and a text-based guideline extractor, thereby increasing the success rate for a given task and improving multimodal grounding with simplified input representation.

[0026] In addition, the method and system for providing a web navigation service based on an action agent according to an embodiment of the present invention accumulates the inference process of demo data and trains small language models (LMs) to be specialized for the domain, thereby performing repetitive tasks on behalf of humans, improving work productivity, and having the effect of utilizing accumulated data over the long term.

[0027] In addition, the method and system for providing web navigation services based on an action agent according to an embodiment of the present invention have the effect of increasing the speed of providing answers by increasing the achievement rate of all processes included in the task and reducing the number of multi-turns by detecting interaction elements that are not visible or hidden on a web page and performing an action.

[0028] In addition, the method and system for providing web navigation services based on an action agent according to an embodiment of the present invention have the effect of significantly increasing the efficiency of human and material resources required to verify license suitability for each data by automatically verifying license suitability for demo data.

[0029] However, the effects obtainable from the present invention are not limited to those mentioned above, and other unmentioned effects can be clearly understood from the description below.

[0030] FIG. 1 illustrates an example of a block diagram of a computing system implementing a web navigation service based on an action agent according to an embodiment of the present invention.

[0031] FIG. 2 illustrates an example of a block diagram of a computing device, which is one of the components of a computing system implementing a web navigation service based on an action agent according to an embodiment of the present invention.

[0032] FIG. 3 illustrates an example of a block diagram in another aspect of a computing device, which is one of the components of a computing system implementing a web navigation service based on an action agent according to an embodiment of the present invention.

[0033] FIG. 4 is a flowchart illustrating a web navigation provision service based on an action agent according to an embodiment of the present invention.

[0034] FIG. 5 is an example of a drawing for explaining a demo set according to an embodiment of the present invention.

[0035] FIG. 6 is an example of a drawing for explaining the machine learning learning flow of an intent extraction model according to an embodiment of the present invention.

[0036] FIG. 7 is a flowchart illustrating a method for performing an action determination process according to an embodiment of the present invention.

[0037] FIG. 8 is an example of an interaction element extracted by analyzing a first web page according to an embodiment of the present invention.

[0038] FIG. 9 is a table explaining the classification criteria for license classes according to an embodiment of the present invention.

[0039] The present invention is capable of various modifications and may have various embodiments; therefore, specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the present invention, and the methods for achieving them, will become clear by referring to the embodiments described in detail below together with the drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various forms. In the following embodiments, terms such as "first," "second," etc., are used not in a limiting sense but for the purpose of distinguishing one component from another. Furthermore, singular expressions include plural expressions unless the context clearly indicates otherwise. Also, terms such as "include" or "have" mean that the features or components described in the specification exist, and do not preclude the possibility that one or more other features or components may be added. Additionally, in the drawings, the size of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily depicted for convenience of explanation, so the present invention is not necessarily limited to what is illustrated.

[0040] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components are given the same reference numerals, and redundant descriptions thereof will be omitted.

[0041]

[0042] FIG. 1 illustrates an example of a block diagram of a computing system that performs a web navigation provision service based on an action agent according to an embodiment of the present invention.

[0043] Referring to FIG. 1, a computing system (1000) that performs a web navigation provision service based on an action agent according to one embodiment of the present invention includes a user computing device (110), a training computing system (150), and a server computing system (130), and each device and system is connected to communicate through a network (170).

[0044] According to an embodiment of the present invention, 1) a user computing device (110) can perform a web navigation provision service based on an action agent by using a local or / and external machine learning model (120) or a machine learning model (140) provided by a server.

[0045] Additionally, according to another embodiment of the present invention, 2) a server computing system (130) communicating with a user computing device (110) may provide a web navigation service based on an action agent to the user computing device (110) on an application or / and the web in response to a request from a user through the user computing device (110).

[0046] In addition, according to another embodiment of the present invention, 3) a user computing device (110) and a server computing system (130) may provide a web navigation service based on an action agent to a user by performing at least a part of the method of providing a web navigation service based on an action agent in conjunction with each other.

[0047] Additionally, according to various embodiments of the present invention, a user computing device (110) and / or a server computing system (130) may learn a machine learning model (120 / 140) that is performed in a method of providing web navigation services based on an action agent through interaction with a training computing system (150) that is communicatedly connected via a network (170). In this case, the training computing system (150) may be separate from the server computing system (130) or may be part of the server computing system (130).

[0048] In some embodiments, the training computing system (150) may be part of the server computing system (130) or part of the user computing device (110).

[0049] In the following description, the user computing device (110) is connected to a server computing system (130) to execute a web navigation service based on an action agent, and the server computing system (130) provides a web navigation service based on an action agent either directly or by using a language model from another server.

[0050] However, it can be understood that cases where part of the process described as being performed in a server computing system (130) is performed in a user computing device (110) are naturally included in the description of the present invention.

[0051] - User Computing Device (110: User Computing Device)

[0052] The user computing device (110) may include all other types of computing devices, such as a smartphone, a mobile phone, a digital broadcasting device, a PDA (personal digital assistants), a PMP (portable multimedia player), a desktop, a wearable device, an embedded computing device and / or a tablet PC.

[0053] Additionally, in the embodiment, the user computing device (110) may further include a predetermined server computing device that provides a web navigation service environment based on an action agent.

[0054] This user computing device (110) includes at least one processor (111) and memory (112).

[0055] Here, the processor (111) may be composed of at least one of a central processing unit (CPU), a graphics processing unit (GPU), ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions, or a plurality of electrically connected processors.

[0056] The memory (112) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof, and may include web storage of a server that performs memory storage functions on the internet. This memory (112) may store data and instructions necessary for the operation of an application for the at least one processor (111) to perform a web navigation provision service based on an action agent.

[0057] In one embodiment, the user computing device (110) can perform various deep learning for web navigation services based on action agents by linking with a deep-learning neural network.

[0058] Here, the deep learning neural network according to the embodiment may include a Convolutional Neural Network (CNN), R-CNN (Regions with CNN features), Fast R-CNN, Faster R-CNN, Mask R-CNN, etc., and may include any deep learning neural network that includes an algorithm capable of performing the embodiments described below, and the embodiments of the present invention do not limit or restrict such deep learning neural networks themselves.

[0059] At this time, according to the embodiment, the deep learning neural network may be installed directly on the server computing system (130) or operate as a separate device from the server computing system (130) to perform deep learning for a web navigation service based on the action agent.

[0060] Additionally, in one embodiment, the user computing device (110) may store at least one machine learning model (120). For example, the user computing device (110) may be various machine learning models, such as a plurality of neural networks (e.g., deep neural networks) that perform a web navigation provision service based on action agents based on structured / quantitative data, or other types of machine learning models including non-linear models and / or linear models, and may be configured as a combination thereof.

[0061] For example, machine learning models may include linear regression, decision trees, random forests, gradient boosting pre-trained language models or / and deep learning models. And neural networks may include at least one of feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks or / and other forms of neural networks.

[0062] Specifically, in the embodiment, the user computing device (110) can store an actionable agent (hereinafter, action agent (AA)) that performs an action by identifying the intent for an input task while interacting with a given environment (specifically, a web page).

[0063] In an example, an action agent (AA) may refer to a computing system that learns user behavior patterns based on a large-scale behavior model (LAM), predicts user behavior when a task is given, and autonomously performs a predetermined action such as clicking, selecting, or dragging. In an example, a user may refer to a person who performs input through an application / program to receive web navigation services.

[0064] Such action agents (AA) can perform web navigation provision services based on the aforementioned machine learning models. In an embodiment, the machine learning models may include an intent extraction model, a guideline extraction model, and / or a license extraction model.

[0065] Additionally, the user computing device (110) may store a model to be used in each process and a prompt template that serves as the basis for input to the model in order to perform at least part of the process for a web navigation provision service based on an action agent through a large language model (LLM) and / or a large behavior model (LAM).

[0066] For example, the user computing device (110) may store 1) a prompt for generating a query from user input, 2) a prompt for analyzing a domain (or web page), 3) a prompt for generating a trajectory, etc.

[0067] That is, in one embodiment, the user computing device (110) can perform a web navigation service based on an action agent based on data received by requesting the execution of some execution steps in a method of providing a web navigation service based on an action agent through a prompt or the like to a language model of an external server.

[0068] In another embodiment, regarding the method of providing a web navigation service based on an action agent requested through a user computing device (110), the server computing system (130) may perform the web navigation service based on the action agent through at least one machine learning model (140) and a machine learning model of another server to provide data to the user computing device (110).

[0069] Such a user computing device (110) may include at least one input component (121) that detects user input. Specifically, the input component (121) may include a sensor system including an image sensor, a position sensor (IMU), an audio sensor, a distance sensor, a proximity sensor, a contact sensor, etc.

[0070] For example, the user input component (121) may include a touch sensor (e.g., a touch screen or / and a touch pad, etc.) that detects a touch of the user's input medium (e.g., a finger or a stylus), an image sensor that detects the user's motion input, a microphone that detects the user's voice input, a button, a mouse and / or a keyboard, etc.

[0071] Here, the image sensor may include an image processing module. Specifically, the image sensor may process still images or video obtained by an image sensor device (e.g., CMOS or CCD).

[0072] In addition, the image sensor can process a still image or video acquired through the image sensor device using an image recognition process (e.g., OCR, etc.) and / or an image processing module to extract necessary information and transmit the extracted information to a processor.

[0073] Additionally, the input component (121) can receive input from an external controller (e.g., mouse, keyboard, etc.) based on an interface module, and in this case, may include an external output device (e.g., speaker).

[0074] At this time, the interface module may be configured to include at least one of a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module, an audio I / O (Input / Output) port, a video I / O (Input / Output) port, an earphone port, a power amplifier, an RF circuit, a transceiver, and other communication circuits.

[0075] In addition, the external output device may include a display system that outputs various information related to a web navigation service based on an action agent as a graphic image.

[0076] Such a display system may be implemented by including at least one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, a 3D display, and an e-ink display.

[0077] Meanwhile, the user computing device (110) including the above-described components may further perform at least some of the functional operations performed by the server computing system (130) described later.

[0078] - Server Computing System (130: Server Computing System)

[0079] The server computing system (130) can perform a series of processes to provide web navigation services based on action agents.

[0080] In detail, in an embodiment, the server computing system (130) can provide a web navigation service based on an action agent by exchanging data necessary to enable a web navigation service process based on an action agent to be executed on an external device, such as a user computing device (110).

[0081] More specifically, in an embodiment, the server computing system (130) can provide an environment in which an application can run on a user computing device (110).

[0082] To this end, the server computing system (130) may include an application program, data and / or instructions, etc. for the application to operate, and may transmit and receive various data based thereon with the external device.

[0083] Additionally, the server computing system (130) includes at least one processor (131) and memory (132). Here, the processor (131) may be composed of at least one or a plurality of electrically connected processors among a central processing unit (CPU), a graphics processing unit (GPU), ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions.

[0084] And the memory (132) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory device, magnetic disk, etc. and combinations thereof. This memory (132) may store data and instructions required for prompt templates, machine learning models (140), etc., for the processor (131) to perform tasks through the language model of the server computing system (130) or / and the language model of an external server.

[0085] For example, a server computing system (130) may include a neural network or / and other multi-layer non-linear models as a machine learning model (140). Exemplary neural networks may include a feed-forward neural network, a deep neural network, a recurrent neural network, and a convolutional neural network.

[0086] In one embodiment, the server computing system (130) may be implemented to include at least one computing device. For example, the server computing system (130) may be implemented to operate a plurality of computing devices according to a sequential computing architecture, a parallel computing architecture, or a combination thereof. Additionally, the server computing system (130) may include a plurality of computing devices connected via a network.

[0087] In an embodiment, the server computing system (130) may further include a data store computing system (1000) (hereinafter, data store) which is a storage for continuously storing and managing raw data (e.g., log data, demo data, etc.) that forms the basis of a method (service) for providing a web navigation service based on an action agent. This data store may include various forms of data storage, ranging from file systems to cloud storage.

[0088] For example, a data store may include at least one database among a relational database that uses a structured query language (SQL) to define and manipulate data, a NoSQL database designed for flexibility and scalability to process unstructured and semi-structured data, a data warehouse optimized for querying and analysis by centralizing large volumes of data from multiple sources as a system used for reporting and data analysis, a data warehouse that stores large volumes of raw data in basic formats such as structured data, semi-structured data, and unstructured data, and a local storage device or Network Attached Storage (NAS) that stores data in files in a format generally accessible by a computer operating system.

[0089] - Training Computing System (150: Training Computing System)

[0090] A training computing system (150) includes at least one processor (151) and a memory (152). Here, the processor (151) may be composed of at least one or a plurality of electrically connected processors, including a central processing unit (CPU), a graphics processing unit (GPU), ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions. The memory (152) may include one or more non-transient / transient computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof. This memory (152) may store data and instructions necessary for the processor (151) to train a machine learning model.

[0091] For example, the training computing system (150) may include a model trainer (160) that trains a machine learning model stored in a user computing device (110) and / or a server computing system (130) using various training or learning techniques, such as back propagation of error.

[0092] For example, the model trainer (160) can perform backpropagation updates to one or more parameters of a machine learning model for a web navigation service based on an action agent based on a defined loss function.

[0093] In some embodiments, performing backpropagation of the error may include performing truncated backpropagation through time. The model trainer (160) may perform a number of generalization techniques (e.g., weight decrement, dropout, knowledge distillation, etc.) to improve the generalization ability of the machine learning model being trained.

[0094] And the model trainer (160) includes computer logic utilized to provide the desired function. The model trainer (160) may be implemented as hardware, firmware and / or software that controls a general-purpose processor. For example, in one embodiment, the model trainer (160) includes a program file stored in a storage device, loaded into memory, and executed by one or more processors. In another embodiment, the model trainer (160) includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium, such as a RAM hard disk or an optical or magnetic medium.

[0095] Networks (170) include, but are not limited to, 3GPP (3rd Generation Partnership Project) networks, LTE (Long Term Evolution) networks, WIMAX (World Interoperability for Microwave Access) networks, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), Bluetooth networks, satellite broadcasting networks, analog broadcasting networks and / or DMB (Digital Multimedia Broadcasting) networks.

[0096] Generally, communication through the network (170) can be performed using any type of wired and / or wireless connection through various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0097]

[0098] FIG. 2 illustrates an example of a block diagram of a computing device, which is one of the components of a computing system (1000) that performs a web navigation provision service based on an action agent according to an embodiment of the present invention.

[0099] Referring to FIG. 2, the computing device (100) included in the user computing device (110), server computing system (130), and training computing system (150) includes a plurality of applications (e.g., applications 1 to N). Each application may include a machine learning library.

[0100] For example, applications may include text messaging applications, virtual keyboard applications, browser applications, chatbot applications, etc.

[0101] In an embodiment, the computing device (100) may include a model trainer (160) for training a machine learning model, and may store and operate the machine learning model to perform a web navigation provision service based on an action agent for input data.

[0102] Each application of the computing device (100) can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In one embodiment, each application can communicate with each device component using an API (e.g., a public API). In one embodiment, the API used by each application may be specific to that application.

[0103]

[0104] FIG. 3 illustrates an example of a block diagram in another aspect of a computing device, which is one of the components of a computing system (1000) that performs a web navigation provision service based on an action agent according to an embodiment of the present invention.

[0105] Referring to FIG. 3, the computing device (200) includes a plurality of applications (e.g., Application 1 to Application N). Each application can communicate with a central intelligence layer. For example, the applications may include a virtual keyboard application, a browser application, a chatbot application, etc. In one embodiment, each application can communicate with the central intelligence layer (and a model stored therein) using an API (e.g., a common API across all applications).

[0106] And the central intelligence layer may include prompts using a plurality of machine learning models or / and language models. For example, as illustrated in FIG. 3, each machine learning model and at least some thereof may be provided for each application and managed by the central intelligence layer. In another embodiment, two or more applications may share a single machine learning model. For example, in some embodiment, the central intelligence layer may provide a single model for all applications. In some embodiment, the central intelligence layer may be included within the operating system of the computing device (200) or otherwise implemented.

[0107] The central intelligence layer can communicate with the central device data layer. The central device data layer may be a centralized data store for the computing device (200). As illustrated in FIG. 3, the central device data layer can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some embodiments, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0108] The technology described herein may refer to servers, databases, software applications, and other computer-based systems, as well as actions taken and information transmitted to or from said systems. It will be recognized that the inherent flexibility of computer-based systems allows for a wide range of possible configurations, combinations, division of tasks, and functionality between and from components. For example, the processes described herein may be implemented using a single device or component or multiple devices or components operating in combination. Databases and applications may be implemented in a single system or in a distributed system across multiple systems. Distributed components may operate sequentially or in parallel.

[0109] - Method for providing a web navigation service based on action agents

[0110] Hereinafter, a method for a computing system (1000) according to an embodiment of the present invention to provide a web navigation service based on an action agent will be described in detail with reference to the attached FIGS. 4 to 8.

[0111] In an embodiment of the present invention, at least one processor of the user computing device (110) may execute at least one application and / or program stored in at least one memory (110) or operate in a background state.

[0112] Hereinafter, the above-mentioned at least one processor operates to execute instructions of the application and performs a web navigation provision service based on the action agent described above, which is described in a shortened manner as being performed by the action agent.

[0113] A web navigation provision service according to an embodiment can be described as a service in which, when a user requests what they want on a web page, an artificial intelligence model navigates the web to determine a route of progress (e.g., a sequence of clicks) to achieve the request, and performs various actions on the web page that the user needs to perform (e.g., clicking, searching, calculation, sending an email, etc.) on their behalf according to the determined route.

[0114] FIG. 4 is a flowchart illustrating a web navigation provision service based on an action agent according to an embodiment of the present invention.

[0115] Referring to FIG. 4, in the embodiment, the action agent (AA) can build a demo set database by creating a demo set. (S101)

[0116] In the embodiments, a demo set may refer to a data set stored by matching the intention and purpose of physical actions performed by actual users on a specific platform (e.g., web pages and / or applications) with the physical actions.

[0117] In the following, physical actions performed by an actual user on a specific platform may be referred to as “actions.” Additionally, for the sake of convenience of explanation, the platform is described as a web page, and accordingly, “actions” may include behaviors such as clicking, scrolling, and dragging.

[0118] This demo set can be information for understanding what intentions a user has and which interaction elements they interacted with on a specific web page.

[0119] In other words, various types of demo sets can be generated for a single web page according to intent, and can be generated for various web pages. Hereinafter, the web pages included in a demo set are referred to as demo pages.

[0120] For example, in the case of a restaurant reservation platform, various log data can be generated according to the user's intent, such as making a restaurant reservation, changing a reservation, searching for a restaurant, or writing a review.

[0121] To generate such a demo set, in the embodiment, the action agent (AA) can obtain log data from the first demo page.

[0122] Here, log data refers to records of all events generated during the operation of a specific platform's system, meaning user behavior-based data.

[0123] In addition, in the embodiment, the action agent (AA) can determine the intent for the acquired log data.

[0124] Here, intent refers to the purpose of the actions performed by a user to achieve a specific task. This intent may be automatically extracted and determined based on a predetermined machine learning model, or it may be extracted and determined from tasks manually entered by the user. In the latter case, the demo set can be saved by matching the tasks to the demo set.

[0125] In other words, when a user performs various actions to accomplish a task on a platform, the intent is to identify the purpose behind each action (e.g., clicking, scrolling, etc.). In this case, if the assigned task is a complex multi-step operation, multiple intents may be included in a single log data.

[0126] Additionally, in the embodiment, the action agent (AA) can match the extracted intent and / or task to the acquired log data.

[0127] That is, in the embodiment, the action agent (AA) can create and save a demo set by mapping the first demo page, log data, intent, and / or task.

[0128] FIG. 5 is an example of a drawing for illustrating a demo set according to an embodiment of the present invention. Specifically, the task illustrated in FIG. 5 is a restaurant reservation task and is part of a demo set that displays the system and user's requests and responses as text.

[0129] Referring to FIG. 5, in the embodiment, the action agent (AA) can combine the first demo page (510), the first intention (520), and / or the first log data (530) to create and store a demo set (DS).

[0130] To create a demo set (DS), the user interacts with the system by performing certain actions on the first demo page (510), and can terminate the task when the final objective is achieved.

[0131] Additionally, when the task is completed, the user can determine the first intention (520) for the actions performed.

[0132] Then, the action agent (AA) can extract first log data (530) that interacted with the system until the task is completed. At this time, the extracted first log data (530) may include data (hereinafter referred to as trajectory) that displays the content of the interaction with the system until the task is completed in text.

[0133] Next, the action agent (AA) can map the first log data (530) including the first demo page (510), the first intention (520), and a plurality of actions (601, 602).

[0134] As illustrated, for example, the first action (601) may be a user input that “provides city information,” and the second action (602) may be a user input that “provides food type.” Additionally, the first intent (520) may be a system response that “searches for a restaurant” based on the city information and food type entered by the user.

[0135] To summarize, the illustrated demo set (DS) may have the meaning that when the first log data (530) is generated through user input selecting city information and food type on the first demo page (510), which is a restaurant reservation platform, the multiple actions (601, 602) included in the first log data (530) are determined to have been performed for the first intention (520) of “searching for a restaurant.”

[0136] In this way, in the embodiment, the action agent (AA) can build a demo set database by storing multiple demo sets generated for multiple web pages.

[0137] The construction of the demo set database according to the embodiment can be seen as establishing a virtual environment-based recording infrastructure by individually accumulating and recording the processes of tasks previously performed on a specific web page.

[0138] Meanwhile, in the embodiment, the action agent (AA) may, when generating a demo set, trace the user's cursor to determine which interaction element located where on the web page an action was performed and match and save the recorded video (hereinafter referred to as the demo image).

[0139] To this end, in the embodiment, the action agent (AA) can prompt engineer based on Set-of-Marks (SoM) Prompting, which highlights the UI where interaction elements exist with bounding boxes and / or numbers, in certain machine learning models trained through a demo set.

[0140] That is, in the embodiment, the action agent (AA) can determine the location information of an interaction element on a web page by using an intent extraction model learned based on prompt engineering and / or demo images displaying the interaction element.

[0141] In other words, at least one machine learning model according to the embodiment may be a machine learning model trained based on a prompt engineering method, but with additional input of a demo image to more accurately identify the existence and location of interaction elements.

[0142] Additionally, in the embodiment, the action agent (AA) can perform machine learning training on at least one machine learning model according to the constructed demo set database. (S103)

[0143] The machine learning model according to the embodiment may include a page analysis model, an intent extraction model, a guideline extraction model, and / or a license extraction model.

[0144] The page analysis model may be a machine learning model that determines detailed information (e.g., location information and / or descriptive information) of interaction elements included in a specific domain (e.g., a specific web page) by image training according to demo images and / or prompt engineering matched to a demo set.

[0145] Since this page analysis model can be absorbed into and operate within the intent extraction model described below, the following explanation is based on the intent extraction model.

[0146] The intent extraction model may be a machine learning model trained to be specialized for a specific domain (e.g., a specific web page and / or a specific task) according to the intent matched to the demo set. Multiple such intent extraction models may be stored in a server computing system (130) separated by domain.

[0147] FIG. 6 is an example of a drawing for explaining the machine learning learning flow of an intent extraction model according to an embodiment of the present invention. Specifically, FIG. 6 illustrates a part of the machine learning learning flow based on a restaurant reservation task.

[0148] In an embodiment, the action agent (AA) can train an intent extraction model to output a plurality of intents included in the demo set using the demo set as input data.

[0149] In this embodiment, the action agent (AA) can support the intent extraction model in learning the locations of interaction elements when extracting multiple intents by adding a demo image (DI) matched to the demo set as input data.

[0150] The demo image (DI) according to the embodiment may be an image that displays actions performed by a user on a predetermined web page as predetermined visual content by tracing them.

[0151] For example, in the demo image (DI), a cursor shape is displayed to show the user's cursor movement, and visual content (e.g., a red circle) that highlights interaction elements that the user has interacted with (e.g., physical actions on the browser such as clicking or dragging) may be displayed.

[0152] That is, by training an intent extraction model based on a demo image, the action agent (AA) according to the embodiment can easily navigate hidden menus and search tabs that are difficult to verify without user actions such as clicking or scrolling. Consequently, all processes included in the task proceed without interruption, increasing the task completion rate and reducing the number of multi-turns required to ask the user back when interrupted, thereby increasing the speed of providing answers.

[0153] In addition, in the embodiment, the action agent (AA) can train an intent extraction model to align multiple extracted intents from lower goals to higher goals using a predetermined algorithm (e.g., top-k algorithm).

[0154] The Top-k algorithm is an algorithm that sorts the actions expected to be performed by the user next in order of probability and selects only the top k tokens. In the embodiment, the action agent (AA) can more effectively remove tokens (actions in the embodiment) with low probability values ​​by using the reinforcement learning algorithm as described above.

[0155] Referring to FIG. 6, in the embodiment, the action agent (AA) can train an intent extraction model to determine an Intent (t), which is a higher goal for a demo set, and to sequentially determine and align lower goals as they move toward Intent (t-1).

[0156] Additionally, in the embodiment, the action agent (AA) can train an intent extraction model to plan the order of actions to be performed sequentially according to intents aligned from lower goals to higher goals.

[0157] In the embodiment, the action agent (AA) can plan the sequence of actions to be performed for each goal to achieve each goal based on this intention extraction model. That is, the action agent (AA) can predict intentions and map actions accordingly.

[0158] In the following, the order of actions to be performed sequentially according to the intentions aligned from lower goals to higher goals may be referred to as a “schedule (SC).”

[0159] That is, in the embodiment, the action agent (AA) can train an intent extraction model to provide a schedule (SC) as output data when a demo set is used as input data. Accordingly, in the embodiment, the action agent (AA) can determine an action to be performed according to the schedule (SC) output from the intent extraction model.

[0160] The guideline extraction model may be a machine learning model trained to provide guidelines that express the schedule (SC) determined according to the task matched to the demo set as text.

[0161] Here, the guideline according to the embodiment may refer to information in which the schedule determined by the intent extraction model for the demo set is displayed as text in a concise natural language format.

[0162] For example, the first guideline may be text in a concise natural language format, such as, “In order for the user to perform the first task on the first web page, the first action was performed first.”

[0163] In an embodiment, the action agent (AA) can take a demo set as input data and train a guideline extraction model to output guidelines for said demo set.

[0164] To this end, in the embodiment, the action agent (AA) can convert the schedule into a guideline. In this case, in the embodiment, the action agent (AA) can convert it using conjunctions and / or context as keywords, such as “when,” “if,” or “should.”

[0165] Additionally, in the embodiment, the action agent (AA) can match the extracted guidelines to the demo set as labels.

[0166] That is, in the embodiment, the action agent (AA) can match the label to the demo set and save it back to the demo set database.

[0167] The guideline extraction model trained in this way can determine the guideline most relevant to the task received from the demo set database when providing a web navigation service, and accordingly, the action agent (AA) in the embodiment can perform an action according to the determined guideline.

[0168] Accordingly, by enabling prior knowledge of prerequisites for future web navigation service provision based on guidelines matched to the demo set, it has the effect of contributing to the action agent (AA) being able to easily plan and execute tasks.

[0169] That is, in the embodiment, the action agent (AA) can provide a web navigation service that performs actions according to a schedule and / or guidelines based on certain machine learning models trained by the method described above.

[0170] Additionally, in the embodiment, the action agent (AA) can acquire a task for the first web page based on user input. (S105)

[0171] Here, the task according to the embodiment may mean an operation that the user wishes to perform on the first web page. Such a task may include text in a natural language format entered by the user.

[0172] For example, the task may include text in natural language format such as, “Find a Western restaurant in OO-dong where two people can dine at 5 o’clock on March 18.”

[0173] To receive such tasks, in the embodiment, the action agent (AA) may provide a web navigation interface that interacts with the user.

[0174] In the embodiments, the web navigation interface may be provided as a chatbot-type widget for a specific web page. For example, the web navigation interface may be provided as a widget within the web page and / or as a separate individual program linked to the web page.

[0175] In an embodiment, the action agent (AA) can understand the language by performing natural language processing (e.g., parsing) based on a predetermined natural language model for the acquired task.

[0176] Meanwhile, in the embodiment, the action agent (AA) may interact with the user by providing a reverse question to the user to specify the task. At this time, if it is determined that the content cannot be performed within the web page where the user requested the task, the action agent (AA) in the embodiment may interact with a third-party website to obtain specific data from a third-party agent and provide it to the user.

[0177] Additionally, in the embodiment, the action agent (AA) can determine a first action according to the acquired task. (S107)

[0178] Specifically, in the embodiment, the action agent (AA) can determine a first action according to the acquired task using a predetermined machine learning model trained based on a pre-built demo set database.

[0179] To this end, in the embodiment, the action agent (AA) may perform an action determination process to determine a first action, which is the first action to be performed according to the acquired task.

[0180] FIG. 7 is a flowchart illustrating a method for performing an action determination process according to an embodiment of the present invention.

[0181] Referring to FIG. 7, in an embodiment, an action agent (AA) can extract intents and guidelines by parsing a task based on a predetermined machine learning model (in an embodiment, an intent extraction model and / or a guideline extraction model). (S301)

[0182] In an embodiment, the action agent (AA) can parse the task using natural language processing techniques such as named entity recognition (NER), semantic role indication (SRL), utterance purpose classification and / or conversation state tracking (DST).

[0183] After analyzing the meaning of the task, the action agent (AA) in the embodiment can extract guidelines for the task based on a guideline extraction model.

[0184] In this case, the guideline extraction model can extract guidelines matched to the action with the highest similarity to the acquired task among the demo set.

[0185] Additionally, in the embodiment, the action agent (AA) can extract at least one intent from the corresponding task based on an intent extraction model.

[0186] Additionally, in the embodiment, the action agent (AA) can align at least one extracted intent from a lower goal to a higher goal.

[0187] In this case, in the embodiment, the action agent (AA) may change the order of the predetermined goals by reflecting the extracted guidelines when at least one intention alignment occurs.

[0188] That is, in the embodiment, the action agent (AA) can identify the user's request (task) by extracting intent and guidelines, understand the intent, and generate a schedule including a rough trajectory for performing the action.

[0189] Additionally, in the embodiment, the action agent (AA) can analyze the first web page to extract interaction elements. (S303)

[0190]

[0191] Specifically, in the embodiment, the action agent (AA) can extract interaction elements by analyzing and / or parsing the HTML structure of the first web page, interaction elements (e.g., buttons, input fields, etc.), layout, etc.

[0192] More specifically, in the embodiment, the action agent (AA) can extract interaction elements by analyzing the first web page based on an intent extraction model trained on a predetermined prompt engineering method (e.g., SoM (Set-of-Marks) Prompting).

[0193] FIG. 8 is an example of an interaction element extracted by analyzing a first web page according to an embodiment of the present invention.

[0194] Referring to FIG. 8, in the embodiment, the action agent (AA) can extract interaction elements included in a screenshot (700) of a first web page based on an intent extraction model.

[0195] Additionally, in the embodiment, the action agent (AA) may display a bounding box (710) that highlights the location of at least one interaction element extracted. Additionally, a box number (720) may be assigned to each bounding box (710).

[0196] Additionally, a text script (800) can be generated and provided that describes in text what kind of interaction element each interaction element assigned a box number (720) is.

[0197] In other words, by visualizing the structure and interaction elements of a web page, the intent extraction model can make more accurate and rapid judgments about the interaction elements present within the web page when identifying trajectories through demo images.

[0198] In other words, in the embodiment, the action agent (AA) can define and provide the extracted interaction elements as a text script (800).

[0199] Additionally, in the embodiment, the action agent (AA) can generate a schedule based on the extracted intent, guidelines, and / or interaction elements. (S305)

[0200] To this end, in the embodiment, the action agent (AA) can determine detailed information (e.g., location information and / or descriptive information) about a plurality of extracted interaction elements.

[0201] At this time, detailed information regarding the interaction element can be determined based on the demo image (DI) and / or text script (800).

[0202] In addition, in the embodiment, the action agent (AA) can generate a schedule based on detailed information regarding the determined interaction element.

[0203] More specifically, regarding a schedule expressed only in general terms, the previously generated schedule can be refined to suit the corresponding web page by identifying what content is displayed on the interaction elements and where they are located based on the detailed information of the interaction elements.

[0204] In addition, in the embodiment, the action agent (AA) can arrange multiple intentions extracted from the first web page in order from lower goals to higher goals based on an intention extraction model.

[0205] When arranging the order of actions that the action agent (AA) intends to perform from the bottom to the top, the action plan established in the first step above may be reflected.

[0206] In addition, in the embodiment, the action agent (AA) can generate a schedule by reflecting the extracted interaction elements in a plurality of aligned intentions.

[0207] In this case, in the embodiment, the action agent (AA) can generate a schedule for the first web page based on a predetermined deep learning model (e.g., zero-shot learning) even if a demo set for the first web page does not exist in the demo set database.

[0208] In this way, in the embodiment, the action agent (AA) can determine the first action to be performed first on the first web page according to the generated schedule. (S307)

[0209] For example, the first action is, “CLICK Salary It is entered and determined in the form of an anchor tag that defines a link in the HTML of the first web page, such as ”, and accordingly, the action agent (AA) can click the text named Salary.

[0210] In this way, in the embodiment, the action agent (AA) can determine the second to nth actions to be performed immediately after the first action included in the schedule.

[0211] At this time, in the embodiment, the action agent (AA) may decide on at least one action, and when the specification of the task is required, stop deciding on and performing the action and provide a reverse question to the user.

[0212] Returning to the example, the action agent (AA) can review the license for the determined first action. (S109)

[0213] Hereinafter, the action agent (AA) reviews the license and provides a web navigation service that prevents copyright disputes, and this is described in a simplified manner as being performed by the license agent (LA). The license agent (LA) according to the embodiment refers to the action agent (AA) that reviews the license and provides a web navigation service that prevents copyright disputes.

[0214] In an embodiment, the license agent (LA) can review the license for the first action based on a license extraction model that has been trained in machine learning in advance.

[0215] The license extraction model may be a machine learning model trained to determine the license suitability of multiple actions included in a trajectory (and / or schedule) matched to a demo set. That is, the license extraction model may be said to exist to comply with the conditions of the corresponding license according to the rules and restrictions on the dataset used by the action agent (AA) when performing a task.

[0216] In this case, the multiple actions subject to license conformity determination may be actions performed using a third-party agent and / or a third-party web page other than the first web page where the user requested the task. For example, they may include a search action through a third-party website or an action of downloading an attached file through the search.

[0217] In an embodiment, the license agent (LA) can train a license extraction model to provide license suitability (in an example, license information) for an input action as output data, using at least one action included in a demo set as input data.

[0218] The license extraction model according to the embodiment can extract license information of at least one action included in the trajectory of the input demo set and / or the sub-data constituting the demo set.

[0219] To this end, the license extraction model can perform searching to extract license information by linking with third-party agents and / or third-party web pages (e.g., Google, arXiv, etc.).

[0220] In this case, the license information according to the embodiment may refer to information regarding usage rights for data included in web pages, domains, files, etc., that are accessed and / or used through actions performed to achieve a task.

[0221] In the embodiments, such license information may include information specified for a plurality of items classified into categories 1 to 4.

[0222] For example, the first category is a data license category, and the first item included in the first category may be the right to modify data and the right to create derivative works, the second item may be the possibility of infringing the original copyright of the output, the third item may be whether rights to the prompt and output are granted, and the fourth item may be whether there is an obligation to notify the data.

[0223] The second category is the personal information and data security category, and the first item included in the second category may be the right to modify data and the right to create derivative works, the second item may be the possibility of infringing the original copyright of the output, the third item may be whether rights to the prompt and output are granted, the fourth item may be whether there is an obligation to notify personal information data, and the fifth item may be whether there is an obligation to notify security data.

[0224] The third category is a data usage period and region category, wherein the first item included in the third category may be a limitation on the data usage period, the second item may be the possibility of revoking the data license grant, the third item may be a limitation on the AI ​​model service period, and the fourth item may be a limitation on the data usage region.

[0225] The fourth category is a legal risk category, wherein the first item included in the fourth category may be the legality of the data collection process, the second item may be a conflict between data licenses, the third item may be a known dispute regarding the AI ​​model in which the data was used, and the fourth item may be the existence of risks in the license agreement.

[0226] That is, the license extraction model can extract license information matching the first action by performing a search on each item of the first to fourth categories for the first action.

[0227] At this time, the license information extracted above may include a “risk score” which is a value calculated through Formula 1 described below.

[0228] [Formula 1]

[0229]

[0230]

[0231]

[0232]

[0233] The above formula The number at the bottom front represents the category number, and the number at the back may represent the item number.

[0234] In the embodiment, the license extraction model can input values ​​into the corresponding formulas according to the search results of each item included in each category.

[0235] At this time, the value entered for each item may be, for example, a value assigned according to a pre-set condition, or a value of 0 or 1 assigned according to Y / N.

[0236] In this way, the license extraction model can input values ​​into each item included in each category and finally calculate the risk score for the first action through the above-mentioned formula 1.

[0237] Here, the license extraction model can determine the license suitability of the first action based on the risk score calculated for the first action.

[0238] Specifically, a determination of license suitability means that an action can be determined as an available license only if the calculated risk score satisfies the pre-set conditions. Conversely, if the pre-set conditions are not satisfied, the action may be determined as an unavailable license.

[0239] For the sake of convenience of explanation, the license conformity determined above is explained based on the dichotomy of classifying and determining usable licenses and / or unusable licenses.

[0240] At this time, the above-mentioned pre-set condition may be set to the risk score for each item, or may be set to the total risk score calculated through the above-mentioned formula 1.

[0241] For example, if the 'Data modification permission item' of the 'Data License Category' has a score of 0, the action may be determined as an unusable license regardless of the scores of other items, according to the conditions set for that item.

[0242] That is, in the embodiment, the license extraction model can provide output data by using at least one action included in the demo set as input data and outputting license information for the received action as one of an available license and / or an unavailable license.

[0243] In this way, in the embodiment, the license agent (LA) can review the license for the first action according to the license information output from the license extraction model.

[0244] Accordingly, the license agent (LA) according to an embodiment of the present invention has the effect of increasing convenience in risk management by constructing a systematic system capable of quantitatively evaluating risks that may occur in various aspects related to data licensing by considering all various risk factors, by calculating risk scores for multiple actions based on a license extraction model.

[0245] Additionally, in another embodiment, the license extraction model may analyze the legal relationship of terms included in at least one action to determine the license information of the first action as an available license and / or an unavailable license.

[0246] That is, the license extraction model can determine the license information of the first action as an available license and / or an unavailable license and provide it as output data.

[0247] Accordingly, in the embodiment, the license agent (LA) can review the license for the first action based on a license extraction model that takes the first action as input data and provides license information for the first action as output data.

[0248] That is, in the embodiment, the action agent (AA) has the effect of increasing the quality of the service by building a data platform that determines license suitability based on a license extraction model, thereby providing a web navigation service that is free from the possibility of license issues even when based on vast amounts of data.

[0249] Additionally, in the embodiment, the action agent (AA) may determine whether to perform the first action based on the result of the license review. (S111)

[0250] Specifically, in the embodiment, the action agent (AA) can perform the first action if the license information for the first action output from the license extraction model is an available license.

[0251] On the other hand, if the license information for the first action is an unusable license, the action agent (AA) in the embodiment may search for another platform to perform the first action, or notify the user that the first action cannot be performed and provide a reverse inquiry to perform another action.

[0252] In this manner, in the embodiment, the action agent (AA) can perform the task by reviewing the licenses for each individual action included in the schedule created to perform the task entered by the user, and if it is determined that all actions are licensed to be available, executing all actions included in the schedule sequentially.

[0253] Meanwhile, in the embodiment, the license agent (LA) can extract a risk score for at least one model utilized while performing a predetermined action included in the task.

[0254] The above model may also include a license extraction model on which a license agent (LA) is based. This is intended to comprehensively manage the licensing risks of all data utilized not only by the license extraction model but also by certain models (hereinafter referred to as "third models") that interact with third-party websites during task execution.

[0255] To this end, in the embodiment, the license agent (LA) can classify license classes for at least one model used when performing a task.

[0256] At this time, the classification criteria for the above license class may be set based on the license information searched during the license review.

[0257] FIG. 9 is a table explaining the classification criteria for license classes according to an embodiment of the present invention.

[0258] Referring to FIG. 9, in the embodiment, the license agent (LA) can classify the first model into classes 1 through 7. At this time, the definition and number of classes classified are not limited to those illustrated.

[0259] That is, in the embodiment, the license agent (LA) can determine the number of data and / or the scope of disclosure of the corresponding model based on the classified license class.

[0260] Accordingly, in the embodiment, the license agent (LA) can store the classified license class, the number of data and / or disclosure scope of the determined model by matching them to the model.

[0261] In addition, in the embodiment, the license agent (LA) can extract a model risk score for each model classified by license class through Equation 2 and Equation 3 described below.

[0262] [Equation 2]

[0263]

[0264] At this time, represents the weight of the i-th data, and represents the number of tokens and / or time of the i-th data, and represents the total number of tokens and / or time of the license extraction model.

[0265] That is, in the embodiment, the license agent (LA) calculates the proportion that the i-th dataset occupies in the corresponding model and divides it by the number of tokens and / or time of the dataset to calculate the weight ( ) can be obtained. These weights may be values ​​that reflect the importance of a specific dataset.

[0266] [Equation 3]

[0267]

[0268] At this time, represents the model risk score, and represents the risk score of the i-th data, and n represents the number of data reflected based on the license classification criteria.

[0269] That is, in the embodiment, the license agent (LA) can obtain a model risk score by adding up all the values ​​obtained by multiplying the risk by dataset and / or the weights of the datasets.

[0270] These model risk scores vary depending on the type of dataset and / or license information, and may reflect the impact each dataset has on the model risk score.

[0271] Additionally, in the embodiment, the license agent (LA) can determine the license suitability of the model based on the acquired model risk score. To this end, there may be a predetermined standard value (condition) to enable the determination of license suitability based on the model risk score.

[0272] In this case, the license conformity determined may mean determining the license as available and / or unavailable for the model, as described in detail in the section on the method for calculating the risk score for the action.

[0273] That is, in the embodiment, the license agent (LA) may determine the model as an available license only if the model risk score satisfies a predetermined condition. Similarly, conversely, if the model does not satisfy a predetermined condition, the license agent may determine the model as an unavailable license and set restrictions on its use.

[0274] For example, a license agent (LA) can filter only the data to be used in the web navigation service by enabling or disabling the availability of an internal model (i.e., a license extraction model) and / or an external model used by the internal model.

[0275] To summarize, in the embodiment, the license agent (LA) obtains model risk scores of license extraction models and / or at least one third model, and can provide results by performing actions on tasks using only models that are available licenses according to the obtained model risk scores.

[0276] Accordingly, the license agent (LA) according to an embodiment of the present invention can easily manage license information for multiple internal and / or external models used during task execution by calculating a model risk score, thereby enabling task execution with a dataset in which the possibility of copyright and / or license disputes is prevented, which has the effect of increasing the time and economic efficiency of data management.

[0277] The method and system for providing a web navigation service based on an action agent according to an embodiment of the present invention has the effect of increasing the success rate of a given task and improving multimodal grounding with simplified input representation by inferring preconditions for an assigned task based on an image-based intent extractor and a text-based guideline extractor.

[0278] In addition, the method and system for providing a web navigation service based on an action agent according to an embodiment of the present invention accumulates the inference process of demo data and trains small language models (LMs) to be specialized for the domain, thereby performing repetitive tasks on behalf of humans, improving work productivity, and having the effect of utilizing accumulated data over the long term.

[0279] In addition, the method and system for providing web navigation services based on an action agent according to an embodiment of the present invention have the effect of increasing the speed of providing answers by increasing the achievement rate of all processes included in the task and reducing the number of multi-turns by detecting interaction elements that are not visible or hidden on a web page and performing an action.

[0280] In addition, the method and system for providing web navigation services based on an action agent according to an embodiment of the present invention have the effect of significantly increasing the efficiency of human and material resources required to verify license suitability for each data by automatically verifying license suitability for demo data.

[0281]

[0282] The embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the computer-readable recording medium may be those specifically designed and configured for the present invention or those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. Hardware devices may be modified into one or more software modules to perform processing according to the present invention, and vice versa.

[0283] The specific embodiments described in this invention are examples and do not limit the scope of the invention in any way. For the sake of brevity of the specification, descriptions of prior electronic configurations, control systems, software, and other functional aspects of said systems may be omitted. Additionally, the connections of lines or connecting members between components shown in the drawings are illustrative of functional connections and / or physical or circuit connections, and may be replaced or additionally represented as various functional connections, physical connections, or circuit connections in actual devices. Furthermore, unless specifically stated as “essential,” “importantly,” etc., a component may not be strictly necessary for the application of the invention.

[0284] Furthermore, although the detailed description of the present invention has been explained with reference to preferred embodiments of the invention, those skilled in the art or those with ordinary knowledge in the relevant technical field will understand that various modifications and changes can be made to the invention without departing from the spirit and technical scope of the invention as set forth in the claims below. Accordingly, the technical scope of the present invention should not be limited to the contents described in the detailed description of the specification, but should be determined by the claims.

[0285] The form for carrying out the invention is the same as the best form for carrying out the invention.

[0286] The present invention has high industrial applicability in that it can be widely utilized to improve productivity and reduce resource consumption in various online service and business process automation (BPA) fields by comprehensively understanding images and text to automate repetitive web-based tasks performed on websites and efficiently building AI models specialized for specific industrial sectors.

Claims

1. A method for a computing system including memory and a processor to provide a web navigation service based on an action agent, wherein A step of constructing a demo set database by creating a demo set that includes at least one action performed for a first intention on a first demo page; A step of performing machine learning training on at least one machine learning model based on the demo set database constructed above; A step of obtaining a task for a first web page based on user input; A step of determining a first action according to an action decision process based on the above-mentioned trained machine learning models; A step of determining whether to perform the first action determined above and generating a schedule in which the first to nth actions are planned; and A step of providing a web navigation service that physically operates the first web page according to the generated schedule; A method for providing web navigation services based on action agents.

2. In Paragraph 1, The step of generating the above demo set is, A step of obtaining first log data including a plurality of actions performed to achieve a predetermined task on a first demo page, and A step of determining a first intention regarding the first log data obtained above, and A step comprising generating a demo set including the first demo page, the first log data, and the first intention, and storing it in a demo set database. A method for providing web navigation services based on action agents.

3. In Paragraph 2, The step of generating the above demo set is, A step of performing prompt engineering to extract and display interaction elements included in a first demo page for a first machine learning model, and A step of matching a first demo image, which is a video recorded by tracing a cursor on the first demo page, to the demo set, and The method further includes the step of performing machine learning training based on the first demo image for the first machine learning model on which the above prompt engineering was performed. A method for providing web navigation services based on action agents.

4. In Paragraph 3, The step of performing prompt engineering to extract and display the above interaction elements is: A step of parsing at least one of the HTML structure, buttons, input fields, and layout of the first web page based on a screenshot of the first web page captured above, and The step of visualizing at least one parsed interaction element A method for providing web navigation services based on action agents.

5. In Paragraph 4, The step of performing machine learning training based on the above demo set database is, The method includes the step of training a first machine learning model using a demo set as input data to provide a plurality of intentions included in the demo set as output data, and The above first machine learning model is, Using a predetermined algorithm, the above-mentioned multiple intentions are sorted from lower goals to higher goals, and Learned to generate and provide the schedule planned to sequentially perform the first to nth actions to achieve the task through the aforementioned aligned plurality of intentions. A method for providing web navigation services based on action agents.

6. In Paragraph 5, The step of performing machine learning training based on the above demo set database is, The method includes the step of training a second machine learning model to provide a guideline, which is information representing the schedule in natural language text format, as output data using a demo set as input data, and The above second machine learning model is, Learned to convert the above schedule into guidelines using conjunctions and context as keywords A method for providing web navigation services based on action agents.

7. In Paragraph 1, The step of acquiring a task for the first web page above is, A step of providing a web navigation interface for a user to input tasks in natural language format, and A step comprising performing a predetermined natural language processing on a task obtained based on the above web navigation interface. A method for providing web navigation services based on action agents.

8. In Paragraph 3, The step of determining the first action according to the above action determination process is, A step of extracting interaction elements by analyzing the first web page based on the first machine learning model on which the above prompt engineering was performed, and A step of determining detailed information including location information and descriptive information for the above-mentioned extracted interaction elements. A method for providing web navigation services based on action agents.

9. In Paragraph 8, The step of generating a planned schedule for the above first to nth actions is A step of extracting intentions and guidelines based on the first and second machine learning models for the above-mentioned acquired task, and A step of generating a schedule according to the extracted intent and guidelines by reflecting detailed information regarding the interaction elements above, and A step comprising performing a first action on the first web page according to the generated schedule. A method for providing web navigation services based on action agents.

10. At least one memory; and It includes at least one processor that reads at least one application stored in the memory and provides a web navigation service based on an action agent; The instructions of the above processor are, Construct a demo set database by creating a demo set that includes at least one action performed for a first intent on a first demo page, and Based on the demo set database constructed above, machine learning training is performed on at least one machine learning model, and Based on user input, obtain a task for the first web page, and Based on the above-mentioned trained machine learning models, a first action is determined according to the action decision process, and Determining whether to perform the first action determined above, and generating a schedule in which the first to nth actions are planned, A web navigation service that physically operates the first web page according to the above-generated schedule A system that provides web navigation services based on action agents.

Images