Image processing device, information processing device, and control method and program therefor

Abstract

This image processing device displays, on a display panel provided to the image processing device, an operation screen relating to a service provided by the image processing device, and assists with settings related to the service through dialogue with a natural language-based generative AI by using a partial region of the operation screen.

Description

Image processing apparatus, information processing apparatus, control methods and programs for the same

[0001] The present disclosure relates to an image processing apparatus, an information processing apparatus, control methods and programs for the foregoing that utilize a natural language model.

[0002] Technologies for giving instructions to an MFP in natural language from sources such as mobile applications (hereinafter abbreviated as "mobile apps") are known. Patent Document 1 proposes a technology that allows a user to instruct an MFP to perform processing via a natural language instruction such as "print this" from a mobile app. Furthermore, services that provide information to users in an interactive natural language format using generative AI such as ChatGPT (registered trademark) are also known.

[0003] Japanese Unexamined Patent Application Publication No. 2014-104625

[0004] According to the above-described conventional technology, it is possible to give natural language instructions to an MFP from a terminal such as a mobile app. However, for processes such as scan transmission processing that require a user to be present at the MFP to execute the job, instructions from a mobile app require switching back and forth between the MFP's operation screen and the mobile terminal's operation screen, which impairs convenience. On the other hand, compared to terminals such as PCs and mobile devices, many MFPs do not support natural language input, which creates the problem that it is difficult to use interactive natural language services.

[0005] The present disclosure has been made in view of at least one of the above-described problems, and provides, for example, a novel mechanism that supports interaction with generative AI in natural language and facilitates favorable device operation.

[0006] For example, the present disclosure relates to an image processing apparatus, comprising: display means for displaying an operation screen related to a service provided by the image processing apparatus on a display panel provided in the image processing apparatus; and control means for supporting setting related to said service through interaction with generative AI in natural language via a partial area of said operation screen.

[0007] Furthermore, this disclosure also includes, for example, an information processing device comprising: display means for displaying an operation screen relating to a service provided by the information processing device; and control means for supporting settings relating to the service through dialogue with a natural language generating AI via a portion of the operation screen, wherein the control means displays one or more candidates for prompts related to setting items of a predetermined service as selectable operation objects in a portion of the operation screen, and generates a prompt according to the selected candidate and inputs it to the generating AI.

[0008] According to this disclosure, for example, it is possible to provide a novel mechanism that supports interaction with a natural language generating AI and facilitates the operation of a device.

[0009] The attached drawings are included in the specification and constitute a part thereof, illustrating embodiments in this disclosure and used to explain the technical ideas derived from this disclosure together with their descriptions. A diagram showing an example of a system for operating the MFP 100 in a natural language dialogue format according to one embodiment. A diagram showing an example of the hardware configuration of the MFP 100 according to one embodiment. A diagram showing an example of the hardware configuration of the language model server 300 according to one embodiment. A sequence diagram for image editing of the MFP 100 according to one embodiment. A flowchart showing the processing procedure of the MFP 100 according to one embodiment. A flowchart showing the processing procedure of the MFP 100 according to one embodiment. A flowchart showing the processing procedure of the language model server 300 according to one embodiment. A diagram showing an example of a screen displayed on the display unit 210 of the MFP 100 according to one embodiment. A diagram showing an example of a screen displayed on the display unit 210 of the MFP 100 according to one embodiment. A diagram showing an example of a screen displayed on the display unit 210 of the MFP 100 according to one embodiment. A diagram showing an example of a screen displayed on the display unit 210 of the MFP 100 according to one embodiment. A diagram showing an example of a prompt sent from the MFP 100 to the language model server 300 according to one embodiment. A diagram showing an example of a prompt sent from the MFP 100 to the language model server 300 according to one embodiment. A sequence diagram for changing the settings of the MFP 100 according to one embodiment. A flowchart showing the processing procedure of the MFP 100 according to one embodiment. A flowchart showing the processing procedure of the MFP 100 according to one embodiment. A flowchart showing the processing procedure of the language model server 300 according to one embodiment. A diagram showing an example of a screen displayed on the display unit 210 of the MFP 100 according to one embodiment. A diagram showing an example of a screen displayed on the display unit 210 of the MFP 100 according to one embodiment. A diagram showing an example of a screen displayed on the display unit 210 of the MFP 100 according to one embodiment. A diagram showing an example of a screen displayed on the display unit 210 of the MFP 100 according to one embodiment. A diagram showing an example of a screen displayed on the display unit 210 of the MFP 100 according to one embodiment. A diagram showing an example of a screen displayed on the display unit 210 of the MFP 100 according to one embodiment. A diagram showing an example of a prompt sent from the MFP 100 to the language model server 300 according to one embodiment.A diagram showing an example of a prompt input screen displayed on the display unit 210 of an MFP 100 according to one embodiment. A diagram showing an example of a prompt input screen displayed on the display unit 210 of an MFP 100 according to one embodiment. A diagram showing an example of a prompt input screen displayed on the display unit 210 of an MFP 100 according to one embodiment.

[0010] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the scope of the claims. While the embodiments describe multiple features, not all of these features are necessary, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0011] <First Embodiment> <System Configuration> The first embodiment of the present disclosure will be described below. Referring to Figure 1, an example of a system for operating the MFP 100 in a natural language dialogue format according to this embodiment will be described. In this system, the MFP 100, which is an image processing device, is connected to the Internet 200 via a network 500, and is further connected to a language model server 300, which is a server device, via the Internet 200. The network 500 may be a wired connection or a wireless connection.

[0012] This disclosure provides support for setting up the MFP 100, which is the target device, in a natural language dialogue format when operating the MFP 100 in such a system configuration. However, this disclosure is not intended to limit itself to this system, and in another embodiment, some functions of the language model server 300 may be provided as functions of the MFP 100. Furthermore, the configuration of this system is not intended to be limited to the device configuration shown in Figure 1, and may include other devices. In this embodiment, the MFP 100 is used as an example of an image processing device, but this disclosure can be applied to any device having at least one of the functions of image forming and image reading. In addition, the language model server 300 may be implemented by a single server device, or by multiple server devices working together to perform processing.

[0013] Furthermore, although this disclosure uses an MFP as an example of the device to be operated on, this disclosure can be applied to any device that has an operating screen. For example, it can be applied to a variety of devices such as information processing devices like PCs, tablets, and smartphones, vehicles such as four-wheeled automobiles, saddle-type vehicles, and railway vehicles, aircraft, drones, ships, robots, and home appliances with operating screens. More specifically, any device to which the technology of this disclosure can be applied is one that can communicate with a language model server and displays an operating screen. The method of inputting prompts to the language model server described below will be described as indirectly inputting natural language by selecting candidate prompts displayed on the operation panel. However, there is no intention to limit this disclosure, and various methods can be applied, such as directly inputting natural language using a keyboard or software keyboard, or using voice input via a microphone.

[0014] <MFP Configuration> Referring to Figure 2, an example of the hardware configuration of the MFP 100 according to this embodiment will be described. The MFP 100 comprises a control unit 110, a communication connector 206, an operation unit 208, a display unit 210, a printing unit 213, and a reading unit 215. The control unit 110 comprises a CPU 201, a ROM 202, a RAM 203, a storage 204, a communication unit I / F 205, an operation unit I / F 207, a UI control unit 209, an image processing unit 211, a printer controller I / F 212, and a scanner controller I / F 214. Each component (each module) can exchange data with each other via the system bus 216.

[0015] The CPU 201 controls the overall operation of the MFP 100. The CPU 201 reads control programs stored in the ROM 202 or storage 204 into the RAM 203 and executes them, performing various controls such as read control and print control. The ROM 202 mainly stores a boot program for deploying control programs that can be executed by the CPU 201, which are stored in the storage 204, into the RAM 203. The RAM 203 is the main memory of the CPU 201 and is used as a temporary storage area for deploying various control programs stored in the work area, ROM 202, and storage 204. Various control programs include, for example, a status collection program 217 and an API execution program 218. In addition to executable control programs, the storage 204 stores print data, image data, various programs, and various setting information.

[0016] In this embodiment, the MFP 100 uses one CPU 201 to execute each process shown in the flowchart described later using one memory (RAM 203), but other configurations are also possible. For example, multiple CPUs, RAMs, ROMs, and storage can work together to execute each process shown in the flowchart described later. Furthermore, some processes may be executed using hardware circuits such as ASICs (Application Specific Integrated Circuits) or FPGAs (Field-Programmable Gate Arrays).

[0017] The control unit I / F 207 connects to the control unit 208. The control unit 208 may include a touch panel input device or push buttons. The scanner controller I / F 214 connects to the reading unit 215. The reading unit 215 reads the image on the sheet and generates image data. The image data generated by the reading unit 215 can be transmitted to an external device or printed on the sheet. The reading unit 215 can also scan multiple sheets by rapidly scanning sheets placed on a document feeder (not shown).

[0018] The printing unit 213 is connected to the printer controller interface 212. Image data to be printed is transferred to the printing unit 213 via the printer controller interface 212. The printing unit 213 receives control commands and image data to be printed, and prints an image based on the image data onto a sheet. The printing method of the printing unit 213 may be electrophotographic or inkjet. In the case of electrophotographic, an electrostatic latent image is formed on a photoreceptor, then developed with toner, the toner image is transferred to a sheet, and the transferred toner image is fixed to form an image. On the other hand, in the case of inkjet, ink is ejected to print an image onto a sheet.

[0019] Furthermore, the MFP 100 is connected to the network 500 via a communication connector 206 connected to the communication unit I / F 205. In addition, the communication unit I / F 205 is connected to the internet 200 on the network 500. The UI control unit 209 is connected to a display unit 210 such as an LCD and controls the display unit. The display unit 210 is an example of a display panel. In this embodiment, a display area for natural language dialogue with the generating AI is displayed on the operation screen, such as an operation screen, which is displayed on the display unit 210 provided on the MFP 100. The display area of ​​such a display unit provided on the MFP 100 is smaller compared to display devices such as PCs. Therefore, one feature of this disclosure is that it is possible to suitably display a dialogue area with the generating AI and provide support for settings, etc., in an environment where the display area is limited. The image processing unit 211 performs rotation, compression, resolution conversion, etc., of image data to be output to the printing unit 213 via the printer controller I / F 212.

[0020] Programs that are loaded into RAM 203 and executed by CPU 201 include a status collection program 217 and an API execution program 218. The status collection program 217 collects status information of each module connected via the system bus 216 and receives and executes setting commands from the PC (described later) via the communication connector 206 and the communication interface 205. The API execution program 218 executes commands received via the communication connector 206 and the communication interface 205.

[0021] <Configuration of the Language Model Server> Referring to Figure 3, the hardware configuration of the language model server 300 according to this embodiment will be described. The language model server 300 includes a CPU 301, ROM 302, RAM 303, storage 304, communication interface 305, and communication connector 306.

[0022] The CPU 301 controls the entire language model server 300. The CPU 301 either directly reads and executes programs held in the ROM 302 and storage 304, or loads them into the RAM 303 before reading and executing them. Based on the control of the CPU 301, the communication interface 305 transmits the data generated by the CPU 301 to the MFP 100 connected to the network 500 via the communication connector 306. The communication interface 305 also receives data from the MFP 100 connected to the network 500 via the communication connector 306. These modules are connected to each other via the system bus 307, enabling them to exchange data.

[0023] The language model program 308, located in RAM 303, is an arbitrary program that returns an appropriate response in natural language from natural language information received from the MFP 100 via the internet 200 through the communication connector 306 and the communication interface 305. The image generation program 309, also located in RAM 303, is an arbitrary program that generates appropriate image data from natural language information received from the MFP 100 via the internet 200 through the communication connector 306 and the communication interface 305.

[0024] <Image Editing Sequence> Referring to Figure 4, an example of the sequence for image editing of the MFP 100 according to this embodiment will be described. In the figure, "S" indicates a step, and the following number indicates the step number. The same applies to subsequent drawings. In the following, for the sake of simplicity, various programs will be described as the main drivers of the processing, but these processes are realized by the CPU reading and executing the programs.

[0025] In S401, the CPU 201 of the MFP 100 detects a button press on the operation unit 208 by the user. In S402, the CPU 201 launches the target application. In this embodiment, an example of a Send application that sends scanned documents to a predetermined destination will be described. In S403, the CPU 201 displays the Send basic screen on the display unit 210.

[0026] In S404, the CPU 201 detects that the user has pressed the scan key 705 via the operation unit 208. In S405, the CPU 201 scans the document from the reading unit 215, generates a preview image in the image processing unit 211, and displays it on the display unit 210. In S406, the CPU 201 generates a sentence (801) for prompt input based on the information read from the ROM 202. In S407, the CPU 201 sends (inputs) the sentence (prompt) created in S406 to the language model server 300 via the communication connector 206. At this time, the transmission is performed using the POST method of HTTP communication. The above prompt is an example of a first prompt.

[0027] In S408, the CPU 301 of the language model server 300 generates (outputs) a response using the language model program 308 based on the text received via the communication connector 306. In S409, the CPU 301 transmits the response text via HTTP communication through the communication connector 306. The response text is shown in 802.

[0028] In S410, the CPU 201 of the MFP 100 generates instruction candidate buttons from the response text received via the communication connector 206. An instruction candidate button is an example of an operation object. In S411, the CPU 201 displays the instruction candidate buttons on the display unit 210. In S412, the CPU 201 detects when the user presses a desired instruction candidate button from among the displayed instruction candidate buttons. In this embodiment, an example is described in which the user presses the important word redaction button 720 from among the displayed instruction candidate buttons, but the user can also select other instruction candidate buttons. In S413, the CPU 201 transmits the instruction text (second prompt) associated with the instruction candidate button pressed by the user and the image data generated in S405 to the language model server 300 via HTTP communication through the communication connector 206. Here, the instruction text is shown as 803.

[0029] This document describes a method for generating instruction text (second prompt) corresponding to the selected instruction candidate button. However, in this disclosure, the content of part of the prompt may be modified. For example, when an instruction candidate button is selected, the corresponding incomplete second prompt may be displayed, and the incomplete part may be filled in based on user input. For example, the incomplete second prompt may be displayed as "Please () the ()", and the text in parentheses may be filled in by user input. In this case, by selecting a predetermined (), one or more candidates to be filled in the parentheses may be displayed in a pull-down format or the like. This allows the user to easily complete the second prompt by selecting and setting a candidate.

[0030] In S414, the CPU 301 of the language model server 300 executes image generation using the image generation program 309 based on the text received via the communication connector 306, saves the generated image in the storage 304, and generates a URL for image download. In this embodiment, the image generation program is described as being stored within the language model server 300, but the technology of this disclosure can also be implemented in a configuration in which an image generation instruction is given via the communication connector 306 to another server capable of executing the image generation program. In S415, the CPU 301 generates a response text including the URL for downloading the generated image using the language model program 308. In S416, the CPU 301 transmits the response text to the MFP 100 via HTTP communication through the communication connector 306. The response text is shown in 804.

[0031] In S417, the CPU 201 of the MFP 100 generates candidate instruction buttons from the response text received via the communication connector 206. In S418, the CPU 201 displays the candidate instruction buttons on the display unit 210. In S419, the CPU 201 detects that the user has pressed the desired candidate instruction button. In this embodiment, an example is described in which the user presses the redaction button 1 730, but the user can also select other candidate instruction buttons. In S420, the CPU 201 sends an image download processing request via HTTP communication to the URL associated with the redaction button 1 730 via the communication connector 206.

[0032] In S421, the CPU 301 of the language model server 300 selects image data from the storage 304 corresponding to the URL requested via the communication connector 306. In S422, the CPU 301 sends the image data via HTTP in response to the request.

[0033] In S423, the CPU 201 of the MFP 100 converts the image data received via the communication connector 206 into an image for display using the image processing unit 211 and displays it on the display unit 210. In S424, the CPU 201 displays the transmit start button 736 on the display unit 210. In S425, the CPU 201 detects that the user has pressed the transmit start button 736. In S426, the CPU 201 transmits the received image data to the specified destination external device via the communication connector 206 and terminates this sequence.

[0034] In this embodiment, the destination setting is assumed to have already been configured. This sequence allows for the easy creation of appropriate prompt instructions according to the application being used, and the execution of instructions for editing scanned images, via the operation unit 208 of the MFP 100, through button display and input processing linked to natural language text. To demonstrate convenience, a sequence for giving instructions to the language model without natural language input is shown, but it is also possible to directly input natural language using soft keyboard processing, etc.

[0035] <Processing Flow of MFP 100> Referring to Figures 5A and 5B, the processing procedure of the MFP 100 according to this embodiment will be described. The processing described below is realized, for example, by the CPU 201 reading programs stored in the ROM 202 and storage 204 into the RAM 203 and executing them. This flowchart starts when power is turned on to the MFP 100. In addition, for the sake of simplicity, various programs executed by the CPU 201 may be described as the main operating entities below.

[0036] In S501, the CPU 201 determines whether or not it has received an application launch operation from the user. If an operation is detected, the process proceeds to S502; otherwise, it returns to S501. In S502, the CPU 201 launches the target application in response to the user's operation, and in S503, the target application screen 701 shown in Figure 7A-1 is displayed on the display unit 210. Details of the target application screen 701 will be described later. The target application screen 701 is an example of an operation screen.

[0037] In S504, the CPU 201 determines whether or not a press of the scan key 705 by the user has been detected. If the CPU 201 detects a press of the scan key 705, it proceeds to S505; otherwise, it returns to S504 and waits until a press of the scan key 705 is detected.

[0038] In S505, the CPU 201 controls the reading unit 215 to scan the document and controls the image processing unit 211 to generate image data. Here, the image data may be generated as a JPEG image, for example. In S506, the CPU 201 converts the generated image data into an image for display using the image processing unit 211 and displays it on the display unit 210. In this embodiment, the CPU 201 controls the image processing unit 211 to convert it into a bitmap with 8-bit depth for each RGB element at a scale corresponding to the liquid crystal of the display unit 210, but this conversion process is not essential.

[0039] In S507, the CPU 201 creates a prompt based on the application executed in S501. The user can register a prompt template for each application, and the registered prompts can be read and used by the CPU 201 from ROM 202 or storage 204. In this embodiment, in S507, a prompt with the contents of 801 is created from the template. Further details will be described later using Figures 8A and 8B. The above prompt is an example of a first prompt.

[0040] In S508, the CPU 201 sends the prompt created in S507 to the language model server 300 via HTTP communication through the communication connector 206, requesting a response from the language model server 300. In S509, the CPU 201 determines whether or not it has received a response from the language model server 300 to the request in S508. If the CPU 201 determines that it has received a response, it proceeds to S510; otherwise, it returns to S509 and waits until it receives a response.

[0041] In S510, the CPU 201 generates candidate buttons based on the received information. In S510, the CPU 201 receives the text content of 802 via the communication connector 206, uses the first part of the bulleted list in 802 as the button name, and generates candidate buttons using the text in parentheses that follows as the prompt input content when selecting a button.

[0042] In S511, the CPU 201 determines whether a candidate button for instruction from the user has been selected. If the CPU 201 detects an operation, it proceeds to S512; otherwise, it returns to S511 and waits until a candidate button for instruction is selected. In S512, the CPU 201 sends a prompt (second prompt) corresponding to the candidate button for instruction selected in S511 to the language model server 300 via HTTP communication through the communication connector 206, and requests a response from the language model server 300. In this embodiment, in S512, the CPU 201 sends the text content of 803 and the image data generated in S505 as input information.

[0043] In S513, the CPU 201 determines whether it has received a response from the language model server 300 to the request in S512. If the CPU 201 determines that it has received a response, it proceeds to S514; otherwise, it returns to S513 and waits until it receives a response. In S514, the CPU 201 generates instruction candidate buttons according to the received information. In this embodiment, in S514, the CPU 201 receives the text content of 804, uses the first part of the bullet points in 804 as the button name, and generates instruction candidate buttons using the following URL as the download destination URL when the button is selected.

[0044] In step S515, the CPU 201 determines whether an instruction candidate button has been selected by the user. If the CPU 201 detects the operation, the process proceeds to S516; if no operation is detected, the process returns to S515 and waits until the instruction candidate button is selected. In step S516, the CPU 201 issues a GET request via HTTP communication to the download URL linked to the instruction candidate button through the communication connector 206. In step S517, the CPU 201 determines whether image data has been received. If the CPU 201 determines that image data has been received, the process proceeds to S518; if it determines that no image data has been received, the process returns to S517 and waits until image data is received. Here, it is assumed that the image data is received as a PDF file.

[0045] In step S518, the CPU 201 converts the image data received by the image processing unit 211 into a display image and displays the image on the display unit 210. Here, it is assumed that the image data is converted into a bitmap with an 8-bit depth for each RGB element at a scale corresponding to the liquid crystal display of the display unit 210. In step S519, the CPU 201 determines whether there is an instruction from the user to acquire another execution result. If the CPU 201 determines that such an instruction has been received, the process returns to S512, and the instruction sentence "Output another execution result" is added to the input prompt having the same content as the text of 803. Thereafter, the CPU 201 executes the processing from step S512 described above again. On the other hand, if the CPU 201 determines that there is no such instruction, the process proceeds to step S520. Note that the CPU 201 can determine that there is no instruction from the user to acquire another execution result, for example, based on the absence of an instruction input for a certain period of time or the pressing of a confirmation button (not shown).

[0046] In step S520, the CPU 201 determines whether there is a transmission instruction from the user. If the CPU 201 determines that a transmission instruction has been detected, the process proceeds to S521; if no transmission instruction is detected, the process returns to S520 and waits until a transmission instruction from the user is detected. In step S521, the CPU 201 transmits the image instructed in S520 to the designated destination via the communication connector 206, and ends the processing of this flowchart. Note that in the present embodiment, the destination is separately designated by the user.

[0047] <Processing Flow of Language Model Server 300> Referring to FIGS. 6A and 6B, a processing procedure of the language model server 300 according to the present embodiment will be described. The processing described below is implemented, for example, by the CPU 301 reading a program stored in the ROM 302 or the storage 304 into the RAM 303 and executing the program. This flowchart starts when power is turned on to the language model server 300. Note that, for ease of explanation in the following description, various programs executed by the CPU 301 may be described as the operation subject in some cases.

[0048] The flowchart of FIG. 6A shows a processing procedure when a prompt is input to the language model server 300 from the CPU 201 of the MFP 100 via the network 500. In S601, the CPU 301 determines whether an input prompt has been received from the MFP 100. If the CPU 301 determines that an input prompt has been received, the process proceeds to S602; if it determines that no input prompt has been received, the process waits until an input prompt is received. In S602, the CPU 301 controls the language model program 308 to determine whether the received prompt includes an image generation instruction. If the CPU 301 determines that an image generation instruction is included, the process proceeds to S603; if it determines that no image generation instruction is included, the process proceeds to S604.

[0049] In S603, the CPU 301 controls the image generation program 309 to cause the image generation to be performed based on instruction information using natural language and an image file related to the instruction. Furthermore, the CPU 301 stores the generated image data in the storage 304 in association with randomly generated URL data. In S604, when URL data is stored in the storage 304, the CPU 301 combines the URL data with a response sentence (804) output by the language model program 308, transmits the combined data to the MFP 100 via the network 500, and sends a response. When the transmission is completed, the processing of this flowchart is ended.

[0050] The flowchart in Figure 6B shows the processing procedure when the language model server 300 receives an image download request from the CPU 201 of the MFP 100 via the network 500. This flowchart starts when the language model server 300 is powered on. An image download request indicates that there is a GET request via HTTP communication for a URL corresponding to image data. In S605, the CPU 301 determines whether or not it has received an image download request from the MFP 100. If the CPU 301 determines that it has received an image download request, it proceeds to S606; otherwise, it returns to S605 and waits until it receives an image download request. In S606, the CPU 301 retrieves the image data associated with the requested URL data from the storage 304, sends it to the MFP 100 via the network 500, and responds to the received request. Once the response is complete, the processing in this flowchart ends.

[0051] <Screen Examples> Referring to Figures 7A-1 to 7B-2, an example of a screen displayed on the display unit 210 by the CPU 201 of the MFP 100 according to this embodiment will be described. Figure 7A-1 is the target application screen 701 displayed in S411. The target application screen 701 is the state after the Send basic screen is displayed in S403, a scan is performed according to the scan instruction and a preview is displayed, and after a prompt is entered to the language model server 300, instruction candidate buttons are generated and displayed. The target application screen 701 includes the application name 702, a login button 703, a preview screen area 704, a scan key 705, a color mode setting button 706, a resolution setting button 707, a mode setting button 708, and a Plex setting button 709. It also includes a Sharpness setting button 710, a FitToPage setting button 711, a format setting button 712, a file name setting button 713, a data volume setting button 714, a 90-degree rotation button 715, and a page delete button 716. It also includes a page selection Back button 717, a page selection Next button 718, a prompt display area 719, a button to redact important words 720, a skew correction button 721, a button to suggest file name candidates 722, and an image quality priority button 723. Furthermore, it is configured to include a reduce data size button 724, a setting button 725 suitable for the Electronic Bookkeeping Law, an other setting button 726, a start key 727, and a stop key 728.

[0052] The application name 702 is information indicating the name of the launched application (target application). The login button 703 is a button that the user presses when logging into the MFP 100, and when pressed, a login screen (not shown) is displayed on the display unit 210. The preview screen area 704 is an area for previewing the state of the scanned image before transmission, and is the area where the scanned image is displayed. The scan key 705 is a button that the user presses when instructing the MFP 100 to perform a document scan, and when pressed, the CPU 201 controls the reading unit 215 to perform document reading.

[0053] The various setting buttons 706 to 714 display the setting name and the current set value. When the user presses each button, the screen transitions to a setting value change screen (not shown) for changing the corresponding setting. The color mode setting determines whether the document is scanned in color or monochrome during scanning. In the diagram, it is shown set to Auto. In this case, the CPU 201 determines from the scanning information whether the document is in color or not, and performs scanning in color if the document is in color, or in monochrome if the document is not in color. The resolution setting is the setting for the vertical and horizontal dpi (dots per inch) of the data transmitted by the MFP 100. A higher resolution setting value generates a higher resolution and sharper image, while a lower value generates a lower resolution and coarser image. In the diagram, 300 dpi vertically and 300 dpi horizontally are set. The mode setting determines the image processing mode used during image generation. In the diagram, it is set to TextImage mode, and CPU 201 performs image processing appropriate for text and images, respectively. Other mode settings include Text mode, which is suitable for text only, and Photo mode, which is suitable for photographs.

[0054] The Plex setting determines whether the scan is performed on one side or both sides of the document. The diagram shows the setting to scan only one side of the document. The Sharpness setting determines the sharpness of the generated image. A higher Sharpness value results in a more rapid change in the density of the outline boundaries of the output image, while a lower value results in a more gradual change in the density of the image boundaries. In the diagram, it is set to 5, which is the midpoint of the 10 setting values, indicating that the sharpness has not been changed. The FitToPage setting determines whether the scanned image data is enlarged or reduced. In the diagram, it is set to On, allowing the image to be enlarged or reduced to a fixed size that is set separately. The Format setting determines the format in which the scanned image data is converted and transmitted. The diagram shows an example where PDF is set, but the user can also select other formats such as JPEG, TIFF, and PDFFA. The File Name setting determines the name to be given to the file generated by CPU201. In the diagram, the file name is created with the predetermined string and date, sample_2024 / 12 / 10, but the user can set the name arbitrarily. The data volume setting indicates the degree of data compression, and in the diagram, image quality is prioritized. In this case, CPU 201 performs data compression to the extent that image quality is not compromised. The user can also set other options to prioritize data volume.

[0055] The 90-degree rotation button 715 rotates the image displayed in the preview screen area 704 90 degrees to the right. When the 90-degree rotation button 715 is pressed, the image displayed in the preview screen area 704 is rotated 90 degrees to the right and displayed. Each time the 90-degree rotation button 715 is pressed, the image rotates 90 degrees, and after pressing it four times, it rotates 360 degrees and returns to its original angle. The page delete button 716 deletes the image displayed in the preview screen area 704 from among the multiple documents scanned by the CPU 201. When the page delete button 716 is pressed, the image currently displayed in the preview screen area 704 is deleted. After the CPU 201 deletes a page, the pages before or after the deleted page are displayed in the preview screen area 704. The page selection Back button 717 displays the image of the previous page of the document displayed in the preview screen area 704. When pressed, the image of the page before the currently displayed image is displayed in the preview screen area 704. The Page Selection Next button 718 is a button that displays the image one page after the image of the document currently displayed in the preview screen area 704. When pressed, the image one page after the currently displayed image is displayed in the preview screen area 704.

[0056] The prompt display area 719 is an area that supports user settings in a natural language dialogue format using generation AI. The prompt display area 719 displays responses from the language model server 300 and requests to the language model server 300 in natural language in the form of buttons. Here, an example of displaying prompt candidates in the form of buttons is described, but there is no intention to limit this disclosure, and for example, a form in which natural language is directly input may also be used. In this case, a method of inputting using a software keyboard displayed on the display unit or a voice input method may be applied.

[0057] Actions that can be performed by user operations on the target application screen 701, such as changing settings, can also be performed using the prompt display area 719. Furthermore, the user can use the prompt display area 719 to use the functions of the language model server 300 to convert the scanned image into a form desired by the user (correct the image). The buttons 720 to 726 displayed in the prompt display area 719 are buttons for inputting information into the prompt, and when pressed by the user, the prompt associated with the button name is sent to the language model server 300. Each of the buttons 720 to 726 is related to a setting item that can be set on the target application screen 701. The important word redaction button 720 is a button for the user to request the redaction of important words in the scanned image and to obtain the image after the redaction process. Here, redaction is a function that hides text strings etc. contained in the scanned image by covering them with ink (black, etc.). The skew correction button 721 is a button for the user to request the skew correction process for the scanned image and to obtain the image after skew correction. The "Suggest File Names" button 722 is used by the CPU 201 to obtain suggested file names based on the scanned image. The "Prioritize Image Quality" button 723 is used to obtain suggested parameter sets for improving image quality. A parameter set is a set of combinations of one or more setting values ​​set by the various setting buttons 706. The "Reduce Data Size" button 724 is used by the CPU 201 to obtain suggested parameter sets for reducing data size. The "Settings for Electronic Bookkeeping Law" button 725 is used by the CPU 201 to obtain suggested parameter sets that comply with the Electronic Bookkeeping Law. The "Other Settings" button 726 is used by the CPU 201 to obtain information for buttons other than 720-725. By pressing this button, a new button may be displayed in place of buttons 720-725, or additional buttons may be displayed in a scrolling format.

[0058] The start key 727 is a button that allows the CPU 201 to send the image displayed in the preview screen area 704 to a pre-set destination. The stop key 728 is a button that cancels any processing currently being performed by the MFP 100's CPU 201, such as sending image data or executing a scan.

[0059] Figure 7A-2 shows the target application screen 701 displayed in S418, and includes a redaction 1 button 730, a redaction 2 button 731, and an other setting button 732. Figure 7A-2 shows that when the redaction 720 for important words in Figure 7A-1 is pressed in S412, the text registered to the button is displayed on the display unit 210. Subsequently, the CPU 201 sends the displayed text to the language model server 300. In this embodiment, the text associated with the button is: "Please suggest several items to redact in the scanned image and output the URLs of the redaction items and image processing results in a bulleted list. #Output example: Redaction 1: URL 1, Redaction 2: URL 2." Based on the response text obtained from the language model server 300, the redaction 1 button 730 and the redaction 2 button 731 are displayed. When the user presses the "Redacted 1" button 730, a download request is executed for the associated URL 1. When the user presses the "Redacted 2" button 731, a download request is executed for the associated URL 2. When the user presses the "Other Settings" button 732, a download request is executed for an image different from the one displayed by the "Redacted 1" button 730 and the "Redacted 2" button 731. When either the "Redacted 1" button 730 or the "Redacted 2" button 731 is pressed, the screen shown in Figure 7B-1 is displayed.

[0060] Figure 7B-1 shows the target application screen 701 displayed at S424, and consists of a redaction image confirmation screen 733, a temporary image preview area 734, a redaction object 735, a send start button 736, an edit confirmation button 737, and a cancel button 738. The redaction image confirmation screen 733 is displayed as a window superimposed on the target application screen 701. The temporary image preview area 734 displays a preview of the image associated with the redaction 1 button 730. The redaction object 735 is the redaction image and is displayed in the position of the image associated with the redaction 1 button 730, but in redaction 2 it is displayed in a different position. In redaction 1, a person's name is judged as an important word, and the redaction object 735 is superimposed only on the person's name. When the send start button 736 is pressed by the user, the image data displayed in the temporary image preview area 734 is sent to the recipient. When the user presses the edit confirmation button 737, the settings are confirmed using the image displayed in the temporary image preview area 734, and preparations are made for display in the preview screen area 704. When the user presses the cancel button 738, the system returns to Figure 7A-2.

[0061] Figure 7B-2 shows the target application screen 701 displayed when the edit confirmation button 737 is pressed in Figure 7B-1, and includes a post-edit confirmation preview image 739. The post-edit confirmation preview image 739 is the same image data as the temporary image preview area 734. When the user presses the send start button 736 at this time, the post-edit confirmation preview image 739 is sent to the destination.

[0062] <Prompt> Referring to Figures 8A and 8B, an example of a prompt generated by the CPU 201 of the MFP 100 according to this embodiment and sent to the language model server 300, and an example of a sentence that the language model server 300 returns to the MFP 100 as a response will be described.

[0063] 801 is an example of a prompt created by running the Scan application in S406. The # at the beginning of the sentence represents a major item, and ## represents a sub-item. In "# Instructions," instructions linked to the application are created, such as "Tell me what settings can be changed and what can be edited on the scanned image." Here, the instructions refer to the content of the instructions to the generating AI and indicate the prompt that will be generated. In "# Output Method," you can specify that the language model server 300 "outputs a bulleted list of executable prompts." In "## Output Format," you can specify the format of the response. Here, as an example of output, it is specified as "Summary Text: Prompt Request Text." Alternatively, you can specify output in a JSON (JavaScript Object Note) format that is easy to use in programs, such as {"Instruction Summary":"Prompt Input Candidates"}. In "## Output Examples," several examples that can actually be presented in the Scan application are listed.

[0064] In "#Prerequisites," you enter the information necessary as a prerequisite for generating the response. The necessary information is specified in the sub-items as follows: "##MFP Capabilities" describes the capabilities of the MFP100, specifically the processing that the MFP100 can perform. Here, it is specified that it supports color, ScantoEmail, PDF, and JPEG, but does not support encrypted PDFs. "##Selectable Standard Sizes" describes the scan sizes that can be specified on the MFP. Here, A5, A4, A3, B5, B4, and LTR are specified as selectable scan sizes. "##Execution Function Name" describes the name of the application executed, which is the type of service provided by the MFP100 selected by the user. Here, it is described as the basic scan function. The above services correspond to the functions provided by the device; for example, if the device is an MFP, it corresponds to print, copy, scan, send, box, facsimile, etc. Therefore, the above services will vary depending on the functionality of the device to which the technology disclosed herein is applied. "## User Execution Button" describes the user action that led to the creation of this prompt. Here, it is described as the start key for the scanning function. "## Default Settings" describes the default parameter set information required to answer the parameter set. Here, the settings for the number of copies, file format, color mode, and data volume are described. "## Setting Description" describes the details of the configurable items for each parameter. Here, a description of the number of copies and file format is provided. Furthermore, as shown in the figure, links to product manual pages can also be included to indicate reference.

[0065] 802 is the response from the language model server 300 to the prompt 801. The response text is generated according to the output format specified in the output method. Here, for ease of explanation, the content is the same as in "## Output Example". In 802, the name of the button to be displayed as a candidate (the name to be displayed on the operation object) and the content of the corresponding prompt are described as one set. A colon ":" is written between the button name and the prompt content to distinguish them from each other. The prompt content is also enclosed in parentheses. Furthermore, a "・" is written at the beginning of each set, and each set is defined separately. Note that these descriptions are examples and do not limit this disclosure. Predetermined characters or symbols can be used for these descriptions.

[0066] 803 is a prompt associated with the button selected by the user (redacting important words 720). The CPU 201 sends a request for redacting along with the scanned image, instructing it to generate several combinations of redacting patterns and resulting images. 804 is the response to 803. It shows that two types of redacted images have been generated, each with a download URL assigned to it. In this embodiment, the language model server 300 has provided the generated image (corrected read image) in the form of a download URL, but it is also possible to send the image data directly.

[0067] As described above, the image processing apparatus according to this embodiment displays an operation screen related to the services provided by the image processing apparatus on a display panel provided on the image processing apparatus, and supports the setting of services through interaction with a natural language generating AI via a part of the operation screen. Furthermore, the image processing apparatus according to this embodiment includes a reading unit that reads images from a document, interacts with the generating AI, and makes changes to the read image generated by reading the document with the reading unit. In this way, prompt input operations on the MFP's operation panel are made easier, and the image data read by the MFP can be easily edited to match the user's intentions based on the response processed by the language model server. In other words, this disclosure can provide a novel mechanism that supports interaction with a natural language generating AI and allows for the appropriate operation of a device.

[0068] <Second Embodiment> <Setting Change Sequence> A second embodiment of the present disclosure will be described below. In the first embodiment described above, an example was given in which an image read from a document is previewed and the read image is changed through dialogue with a natural language generating AI. In this embodiment, an example will be given in which the reading settings are determined or changed through dialogue with a natural language generating AI before performing a scan. Note that the same configuration as in the first embodiment will not be described.

[0069] First, with reference to Figure 9, an example of a sequence for changing the settings of the MFP 100 according to this embodiment will be described. In the figure, "S" indicates a step, and the following number indicates the step number. The same applies to subsequent drawings. In the following, for the sake of simplicity, various programs will be described as the main drivers of the processing, but these processes are realized by the CPU reading and executing the program.

[0070] In S901, the CPU 201 of the MFP 100 detects that the user has pressed the estimate scan button 1208 on the operation unit 208. In S902, the CPU 201 launches the target application. In this embodiment, an example of an estimate scan application that sends scanned documents to a predetermined destination is shown. In S903, the CPU 201 displays the Send basic screen on the display unit 210. In S904, the CPU 201 generates a sentence (801) for prompt input based on the information read from the ROM 202. In S905, the CPU 201 sends the sentence created in S904 to the language model server 300 via the communication connector 206. At this time, the transmission is performed using the POST method of HTTP communication.

[0071] In S906, the CPU 301 of the language model server 300 generates a response using the language model program 308 based on the text received from the communication connector 306. In S907, the CPU 301 transmits the response text via HTTP communication through the communication connector 306. The response text is shown in 1301. Further details will be described later.

[0072] In S908, the CPU 201 of the MFP 100 generates instruction candidate buttons from the response text received via the communication connector 206. In S909, the CPU 201 displays the instruction candidate buttons on the display unit 210. In S910, the CPU 201 detects that the user has pressed the selected instruction candidate button. In this embodiment, the "Reduce data size" button 1212 is used as an example, but other instruction candidates can also be selected. In S911, the CPU 201 sends the instruction text associated with the instruction candidate button pressed by the user to the language model server 300 via HTTP communication through the communication connector 206. Here, the instruction text is shown in 1302. Further details will be described later.

[0073] In S912, the CPU 301 of the language model server 300 generates a response sentence for the parameter set using the language model program 308 based on the text received via the communication connector 306. In S913, the CPU 301 transmits the response sentence via HTTP communication through the communication connector 306. The response sentence is shown in 1303. Further details will be described later.

[0074] In S914, the CPU 201 of the MFP 100 generates candidate instruction buttons from the response text received via the communication connector 206. In S915, the CPU 201 displays the candidate instruction buttons on the display unit 210. In S916, the CPU 201 detects that the user has pressed the selected candidate instruction button. In this embodiment, the low-resolution button 1214 is used as an example, but other candidate instructions can also be selected. In S917, the CPU 201 displays the setting value change screen 1218 shown in Figure 12B-2 on the display unit 210. In S918, the CPU 201 detects that the user has pressed the confirmation button 1221, confirms the setting value change, and makes the change. In S919, the CPU 201 displays Figure 12C-1, which reflects the setting result, on the display unit 210. As shown in Figure 12C-1, the value of the resolution button, 1224, has changed from 300x300 to 100x100.

[0075] At S920, the CPU 201 detects that the user has pressed the scan key 705 on the operation unit 208 of the MFP 100. At S921, the CPU 201 scans the document from the reading unit 215. At S922, the CPU 201 generates a preview image in the image processing unit 211 and displays it on the display unit 210. At S923, the CPU 201 detects that the user has pressed the send start button 736. At S924, the CPU 201 sends the image data to the destination via the communication connector 206. In this embodiment, it is assumed that the destination setting has already been set. This sequence makes it easy to create appropriate prompt instruction sentences according to the application being used and to give instructions to edit scanned images using the operation unit 208 of the MFP 100, through button display and input processing linked to natural language sentences. To demonstrate convenience, a sequence is shown in which instructions are given to the language model without natural language input, but it is also possible to directly input natural language using soft keyboard processing, etc.

[0076] <Processing Flow of MFP 100> Referring to Figures 10A and 10B, the processing procedure of the MFP 100 according to this embodiment will be described. The processing described below is realized, for example, by the CPU 201 reading programs stored in the ROM 202 and storage 204 into the RAM 203 and executing them. This flowchart starts when power is turned on to the MFP 100. For the sake of simplicity, various programs executed by the CPU 201 may be described as the main operating entities below.

[0077] In S1001, the CPU 201 waits for a user to launch an application. If an operation is detected, the process proceeds to S1002; otherwise, it returns to S1001. In S1002, the CPU 201 launches the target application. In S1003, the CPU 201 displays the target application screen 701 on the display unit 210. In S1004, the CPU 201 creates a prompt based on the application executed in S1001. A prompt template can be registered for each application and can be read from the ROM 202 or storage 204. In S1004, a prompt with the contents of 801 is created from the template. In S1005, the CPU 201 requests the prompt created in S1004 from the language model server 300 via HTTP communication through the communication connector 206.

[0078] In S1006, the CPU 201 determines whether it has received a response from the communication connector 206 in response to the request in S1005. If it has received a response, it proceeds to S1007; otherwise, it returns to S1006. In S1007, the CPU 201 generates candidate buttons based on the received information. In S1007, the CPU 201 receives the text content of 1301, and generates candidate buttons using the first part of the bulleted list as the button name and the following text in parentheses as the prompt input content when selecting a button. Details of 1301 will be described later.

[0079] In S1008, the CPU 201 determines whether or not it has received an operation from the user for one of the candidate instruction buttons. If an operation is detected, it proceeds to S1009; otherwise, it returns to S1008. In S1009, the CPU 201 requests the content of the candidate instruction button selected in S1008 as prompt input from the language model server 300 via HTTP communication through the communication connector 206. In S1009, the text content of 1302 is sent as input information. Details of 1302 will be described later.

[0080] In S1010, the CPU 201 determines whether it has received a response from the communication connector 206 in response to the request in S508. If it has received a response, it proceeds to S1011; otherwise, it returns to S1010. In S1011, the CPU 201 generates candidate instruction buttons according to the received information. In S1011, it receives the text content from 1303, and generates candidate instruction buttons using the first part of the bulleted list as the button name and the following string as the setting change content. At this time, the setting change content may be output in a specific format such as JSON. In S1012, the CPU 201 determines whether it has received an instruction from the user to obtain other execution results. If it has received an instruction to obtain other execution results, it returns to S1009 and, in addition to the same prompt input again, adds the instruction "output another execution result". If there is no instruction to obtain other execution results, it proceeds to S1013. In S1013, the CPU 201 waits for the user to operate one of the candidate instruction buttons. If an operation is detected, proceed to S1014; otherwise, return to S1013.

[0081] In S1014, the CPU 201 displays a setting value change screen on the display unit 210 corresponding to the user's instruction candidate button in S1013. In S1015, the CPU 201 determines whether it has received an instruction from the user to change the setting value. For example, if it receives an operation of the confirm button, the CPU 201 determines that it has received an instruction to change the setting value. In S1016, the CPU 201 changes the setting value according to the setting value change content in S1011 and displays it on the display unit 210. In S1017, the CPU 201 determines whether it has detected the user pressing the scan key 705. If an operation is detected, it proceeds to S1018; otherwise, it returns to S1017.

[0082] In S1018, the CPU 201 scans the document using the reading unit 215 and generates image data using the image processing unit 211. Here, it is assumed that JPEG image data is generated. In S1019, the CPU 201 converts the image data into a displayable image using the image processing unit 211 and displays it on the display unit 210. Here, it is assumed that the image is converted into a bitmap with 8-bit depth for each RGB element and scaled according to the liquid crystal display of the display unit 210 and displayed. In S1020, the CPU 201 determines whether it has received a transmission instruction from the user. If a transmission instruction is detected, it proceeds to S1021; otherwise, it returns to S1020. In S1021, the CPU 201 transmits the image instructed in S1020 to the destination via the communication connector 206 and terminates the processing of this flowchart.

[0083] <Processing Flow of Language Model Server 300> Referring to Figure 11, the processing procedure of the language model server 300 according to this embodiment will be explained. The processing described below is realized, for example, by the CPU 301 reading programs stored in the ROM 302 and storage 304 into the RAM 303 and executing them. This flowchart starts when power is turned on to the language model server 300. For the sake of simplicity, various programs executed by the CPU 301 may be described as the main operating entities below. Figure 11 shows the processing procedure when the language model server 300 receives a prompt input instruction from the CPU 201 of the MFP 100 via the network 500.

[0084] In S1101, the CPU 301 determines whether or not it has received a prompt input instruction from the MFP 100. If a prompt input instruction is received, the process proceeds to S1102, where the CPU 301 generates a parameter set using the language model program 308 and proceeds to S1103. The parameter set is a combination of parameters used when executing a job on the MFP 100. In response to the instruction "Tell me the settings to reduce the data size," parameter values ​​prioritizing low resolution and data volume are presented. In S1103, the CPU 301 sends the response text output by the language model program 308 back to the MFP 100 via the network 500, and the processing of this flowchart ends.

[0085] <Screen Examples> Referring to Figures 12A-1 to 12C-2, an example of a screen displayed on the display unit 210 by the CPU 201 of the MFP 100 according to this embodiment will be described. Figure 12A-1 is the home screen 1201, a screen in which buttons for various applications that the MFP 100 can execute are displayed in a row. The home screen 1201 consists of a copy button 1202, a fax button 1203, a scan button 1204, a mobile portal button 1205, a menu button 1206, an address book button 1207, and a quotation scan button 1208. When these buttons are selected, the CPU 201 displays the screen corresponding to each button on the display unit 210 and executes the application or function.

[0086] The copy button 1202 is a button that invokes the copy function. The fax button 1203 is a button that invokes the fax function. The scan button 1204 is a button that invokes the scan function. The mobile portal button 1205 is a button that invokes the mobile connection function. The menu button 1206 is a button that invokes the settings menu function. The address book button 1207 is a button that invokes the address book function. The estimate scan button 1208 is a button that the user has registered as the application name "estimate scan". When the estimate scan button 1208 is pressed, the CPU 201 executes processing with the settings registered by the user. In addition, in this embodiment, not only the name of the estimate scan button 1208 but also the recipient and setting values ​​can be customized and registered by the user. In this embodiment, an example in which the estimate scan button 1208 is selected by the user in S901 will be described.

[0087] Figure 12A-1 shows the estimate scanning application screen 1209 displayed at S909, and includes the application name 1210, a resolution button 1211, and a "Reduce data size" button 1212. The estimate scanning application screen 1209 is an example of an operation screen. Here, components necessary for explaining this embodiment are denoted by reference numerals and described accordingly; other components not denoted by reference numerals are not excluded from this disclosure. The following drawings will be described in the same manner. Note that the screen configuration shown in Figure 12A-2 is the same as the screen configuration shown in Figure 7A-1, and a detailed explanation of similar configurations will be omitted.

[0088] The app name 1210 is information indicating the name of the launched app, and the app name is displayed. The resolution button 1211 is a button that allows you to change the resolution setting, and the current setting value is displayed. The "Reduce data size" button 1212 is a button generated from the response to the prompt, and is the button to input the prompt "Tell me the setting value to reduce the data size".

[0089] Figure 12B-1 shows the application screen displayed in S915. It consists of a prompt input 1213, a low resolution button 1214, a data volume priority button 1215, a batch apply button 1216, and an other button 1217. The prompt input 1213 shows the result of inputting "Tell me the setting value to reduce the data size" which is linked to the "Reduce data size" button 1212. The low resolution button 1214 is a button that changes the resolution to 100x100. The data volume priority button 1215 is a button that changes the data volume setting to the data volume priority setting. The batch apply button 1216 is a button that applies the effects of all the buttons suggested by the low resolution button 1214 and the data volume priority button 1215. The other button 1217 is a button that prompts the user to tell it another setting value.

[0090] Figure 12B-2 shows the setting value change screen 1218 displayed in S917 when the low-resolution button 1214 is pressed. The setting value change screen 1218 consists of a setting change button name 1219, a setting change confirmation message 1220, a confirmation button 1221, and a cancel button 1222. The setting change button name 1219 indicates the name of the button pressed to change the setting. Here, the low-resolution version is displayed because the low-resolution button was pressed. The setting change confirmation message 1220 is a message that shows how the setting will change compared to the original setting. The confirmation button 1221 is a button that, when pressed, reflects the change as a setting. The cancel button 1222 is a button that returns to the original screen without changing the setting.

[0091] Figure 12C-1 shows the screen displayed in S919 when the confirmation button 1221 is pressed. This screen includes a change notification 1223, a resolution button change notification 1224, and a scan button 1225. The change notification 1223 indicates that the resolution has been changed to 100x100. The resolution button change notification 1224 shows that the button display has been changed because the resolution has been changed to 100x100. The scan button 1225 is a button that starts scanning the document when pressed.

[0092] Figure 12C-2 shows the screen displayed in S922 when the scan button 1225 is pressed. This screen consists of a preview image 1226, a start button 1227, and a stop button 1228. The preview image 1226 displays the image scanned from the document using the scan button 1225. The start button 1227 is the button that initiates the transmission of image data to the destination. The stop button 1228 is the button that interrupts the process that is currently running.

[0093] <Prompt> Referring to Figure 13, an example of a prompt generated by the CPU 201 of the MFP 100 according to this embodiment and sent to the language model server 300, and a sentence that the language model server 300 returns to the MFP 100 as a response will be described.

[0094] 1301 is the answer to 801. The answer text is generated in the specified output format, matching the output example. 1302 is a prompt associated with the button selected by the user. Along with the instruction "Please tell me the settings to reduce the data size," an example of the output format is shown. Any output format is acceptable as long as it is in a format that the image forming apparatus can analyze. 1303 is the answer to 802. Possible settings to reduce the data size include lowering the resolution and increasing the compression ratio by prioritizing data volume. A suggestion to apply these settings all at once is also made.

[0095] As described above, the image processing apparatus according to this embodiment includes a reading unit that reads an image from a document, interacts with the generating AI, and determines or changes the reading settings when the reading unit reads the document. In this way, according to this embodiment, prompt input operations on the MFP are made easier, and the settings of the MFP's execution functions can be easily set to match the user's intentions.

[0096] <Modifications> This disclosure is not limited to the above embodiments and can be modified in various ways. For example, the placement of the prompt display area (prompt screen) for interacting with the generated AI using natural language on the application screen can be changed to any position depending on the size and specifications of the display screen of the device on which it is displayed.

[0097] Referring to Figures 14A to 14C, various forms of arrangement and calling methods for the prompt screen (prompt display area 719) will be explained. Figure 14A shows screens 1400, 1410, 1420, and 1430 in which the prompt screen 1402 can be arranged above, below, to the left, or to the right of the application screen 1401. Note that the area sizes of the application screen 1401 and the prompt screen 1402 can also be arbitrarily changed.

[0098] Figure 14B shows an example of a display that is effective, for example, in the case of an MFP with a small screen size for the display unit 210. In an MFP with a small screen size, if the application screen 1401 and the prompt screen 1402 are displayed on the same screen simultaneously, the font size and other elements must be reduced, leading to a decrease in visibility. Therefore, in such cases, as shown in screen 1440, a prompt call button 1403 is placed on the application screen 1401. When the prompt call button 1403 is pressed, the prompt screen 1402 shown in screen 1450 is called and displayed superimposed on the current screen. Note that the prompt call button 1403 is not limited to something displayed on the screen, but may be handled by a hardware key or the like. Also, although the example in screen 1450 shows the prompt screen 1402 being displayed across the entire display unit 210, it is not limited to this, and may be configured to be displayed in a pop-up window or the like, on a part of the display unit 210.

[0099] Figure 14C shows an example of changing the size of the displayed prompt screen, where screen 1460 shows a portion of the prompt screen 1402. The prompt screen 1402 may also be resized as shown in screen 1470 by the user performing touch operations on the prompt screen 1402 (e.g., pinch-out operation, pinch-in operation).

[0100] Here, the prompt screen is described as a single screen, but it may also be displayed using multiple screens. In this case, one screen may display prompt input suggestions, while other screens may display the history of natural language dialogue so far. When the prompt screen is divided into multiple screens, it may be divided into vertical or horizontal areas on the screen, or into vertical and horizontal areas in an L-shape.

[0101] <Other Embodiments> The technical ideas derived from this disclosure are not limited to the exemplary embodiments disclosed, but are intended to encompass various modifications of the exemplary embodiments, or substitutions with equivalent structures or functions. The scope of the following claims should be interpreted in the broadest way to encompass all such modifications and equivalent structures and functions.

[0102] This disclosure is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to make the scope of the invention public.

[0103] This application claims priority based on Japanese Patent Application No. 2025-052439, filed on 26 March 2025, and all of its contents are incorporated herein by reference.

[0104] 100: MFP (Image Processing Unit), 110: Control Unit, 200: Internet, 201: CPU, 202: ROM, 203: RAM, 204: Storage, 205: Communication Interface, 206: Communication Connector, 207: Operation Interface, 208: Operation Unit, 209: UI Control Unit, 210: Display Unit, 211: Image Processing Unit, 212: Printer Controller Interface, 213: Printing Unit, 214: Scanner Controller Interface, 215: Reading Unit, 216: System Bus, 217: Status Collection Program, 218: API Execution Program, 300: Language Model Server, 500: Network

Claims

1. An image processing apparatus comprising: a display means for displaying an operation screen relating to a service provided by the image processing apparatus on a display panel provided on the image processing apparatus; and a control means for supporting settings relating to the service through dialogue with a natural language generating AI via a part of the operation screen.

2. The image processing apparatus according to claim 1, wherein the generated AI is provided by a server that can communicate via a network, and the control means generates a first prompt related to a predetermined service among the services provided by the image processing apparatus and transmits it to the server.

3. The image processing apparatus according to claim 2, wherein the first prompt includes an instruction to be requested to the generating AI, a method and example of outputting the generating AI's response to the instruction, information regarding the capabilities of the image processing apparatus, the type of service selected by the user via the operation screen, the default setting for the type of service, and a description of the setting.

4. The image processing apparatus according to claim 2 or 3, wherein the control means generates one or more candidates for a second prompt related to a setting item of a predetermined service based on the response of the generating AI to the first prompt transmitted to the server, and the display means displays the one or more candidates for the generated second prompt as selectable operation objects in a part of the operation screen.

5. The image processing apparatus according to claim 4, wherein the response of the generating AI includes one or more sets of a name to be displayed on the operation object and the content of a corresponding second prompt.

6. The image processing apparatus according to claim 5, wherein the response of the generating AI is defined by dividing each of the one or more sets by predetermined characters or symbols.

7. The image processing apparatus according to any one of claims 4 to 6, wherein one or more candidates for the second prompt are displayed in a manner that allows modification of part of the content of the prompt.

8. The image processing apparatus according to any one of claims 4 to 7, wherein one or more candidates for the second prompt include setting items relating to the redaction of a portion of the image read by the image processing apparatus, the skew correction of the image read, the file name of the image read, the image quality of the image read, the size of the image read, and the orientation when saving the image read.

9. The image processing apparatus according to claim 8, wherein when a user selects one of the one or more candidates of the displayed second prompt, the control means transmits the second prompt corresponding to the selected candidate and the read image to the server and obtains a corrected read image.

10. The image processing apparatus according to any one of claims 2 to 9, wherein a portion of the operation screen is located at the top, bottom, left, or right of the operation screen.

11. The image processing apparatus according to any one of claims 2 to 10, wherein a portion of the operation screen is displayed superimposed on the operation screen by operating an operation object on the operation screen.

12. The image processing apparatus according to any one of claims 2 to 11, wherein the size of a part of the operation screen can be enlarged or reduced.

13. The image processing apparatus according to any one of claims 2 to 12, wherein the operation screen is the screen of an application for providing services for the image processing apparatus.

14. The image processing apparatus according to any one of claims 2 to 13, wherein the server comprises one or more server devices.

15. An image processing apparatus according to any one of claims 2 to 14, further comprising a reading means for reading an image from a document, wherein the control means interacts with the generating AI and makes changes to the read image generated by reading the document by the reading means.

16. The image processing apparatus according to claim 15, further comprising a transmission means for transmitting the corrected read image to an external device.

17. An image processing apparatus according to any one of claims 1 to 14, further comprising a reading means for reading an image from a document, wherein the control means interacts with the generating AI and determines or changes the reading settings when the reading means reads the document.

18. An information processing device comprising: a display means for displaying an operation screen relating to a service provided by the information processing device; and a control means for supporting settings relating to the service through dialogue with a natural language generating AI via a portion of the operation screen, wherein the control means displays one or more candidates for prompts relating to setting items of a predetermined service as selectable operation objects in a portion of the operation screen, and generates a prompt according to the selected candidate and inputs it to the generating AI.

19. A method for controlling an image processing device, comprising: a display step in which a display means displays an operation screen relating to a service provided by the image processing device on a display panel provided on the image processing device; and a control step in which a control means assists in setting up the service through dialogue with a natural language generating AI via a part of the operation screen.

20. A program for causing a computer to perform each step in a control method for an image processing device, wherein the control method includes a display step in which a display means displays an operation screen relating to a service provided by the image processing device on a display panel provided on the image processing device, and a control step in which a control means assists in setting up the service through dialogue with a natural language generating AI via a part of the operation screen.

21. A control method for an information processing device, comprising: a display step in which a display means displays an operation screen relating to a service provided by the information processing device; and a control step in which a control means supports settings relating to the service through dialogue with a natural language generating AI via a part of the operation screen, wherein the control step displays one or more candidates for prompts related to setting items on the operation screen as selectable operation objects in a part of the operation screen, and generates a prompt according to the selected candidate and inputs it to the generating AI.

22. A program for causing a computer to perform each step in a control method for an information processing device, wherein the control method includes a display step in which a display means displays an operation screen relating to a service provided by the information processing device, and a control step in which a control means supports settings relating to the service through dialogue with a natural language generating AI via a part of the operation screen, wherein the control step displays one or more candidates for prompts relating to setting items on the operation screen as selectable operation objects in a part of the operation screen, and generates a prompt according to the selected candidate and inputs it to the generating AI.

Images