Monitoring system for a communication device

Abstract

The invention relates to a monitoring system for a programmable logic communication device, such as is commonly used, for example, for interaction with human users, for the open-loop and closed-loop control of processes and / or of automated industrial installations. The invention provides a monitoring system with visual output, by means of which the status of the communication device is communicated to the user for the first time in a simple, intuitive manner and by means of clear symbols; in particular, it becomes clear to the user whether a fault is present, whether the communication device is currently processing data, waiting for an input, and more.

Description

[0001] 202505839

[0002] 1

[0003] Description

[0004] Monitoring system for a communication device

[0005] The invention relates to a monitoring system for a programmable logic controller (PLC) communication device, such as those commonly used for interaction with human users, for controlling and regulating processes and / or automated industrial plants.

[0006] Technical background

[0007] AI-based tools and technologies are rapidly spreading across all areas of life and industry. The interaction between user and machine, facilitated by communication devices via inputs such as text, speech, gestures, and / or facial expressions, plays a central role. It is crucial that the machine understands the user and asks clarifying questions when necessary. Furthermore, it is essential that communication occurs within a predictable timeframe and with predictable energy consumption, ensuring that actions initiated, controlled, and / or stopped through communication are implemented promptly.

[0008] A communication device typically comprises modules that address the essential functionalities of a communication system. These modules include, in particular, the following functionalities: Besides simple input devices such as buttons, sliders, touchscreens, microphones with integrated speech processing, AI with corresponding processors for response generation, modules for multimodal interaction, recording devices such as cameras and speakers, devices for processing speech, text, images, and videos, gesture recognition, and finally, connectivity to external systems and / or data sources, APIs for integration into other applications, as well as monitoring and analysis. These modules work together to enable the most natural and intelligent interaction, conversation, and / or communication possible between humans and machines. This includes, for example, the machine recording, or "listening," when the human makes an acoustic input.When so many modules, both hardware and software, interact, the system's susceptibility to malfunctions and / or errors increases. In addition, there is the possibility that the system is simply overloaded and therefore shows no response. 202505839.

[0009] 2

[0010] This involves input and output devices through which user input and machine output occur. Examples include monitors, touchscreens, cameras, microphones, keyboards, and / or speakers. The communication device attempts to interact with the user flawlessly, without errors, and / or in real time, while simultaneously initiating the control process. However, delays and / or errors can occur on the communication device side for various reasons, which the user cannot detect because they occur in the "back end." The problem is that the user, unclear about what is happening in the back end, cannot distinguish errors and / or malfunctions from processing time when the system appears to be stalled. Furthermore, additional input, delays, or actions such as switching the device off and on again can unnecessarily burden the system and cause it to crash. This should be avoided.

[0011] Summary of the invention

[0012] The object of the present invention is to provide a monitoring system for an AI-based communication device that provides the user with information about what is happening on the back end, in particular whether the system is not responding due to a malfunction or error, or whether processes are currently taking place in the back end that simply take time.

[0013] This problem is solved by the subject matter of the present invention as disclosed in the description, figures and claims.

[0014] Accordingly, the present invention relates to a monitoring system with a display device for a programmable logic controller (PLC) communication device, wherein the display device is part of a device suitable for user interaction and comprises at least one graphical user interface (GUI) and one or more interface(s) (API) with the back end of the communication device at the front end, wherein the monitoring system has a monitoring unit with which it detects the data flow and / or the amount of data exchanged between the back end and the front end, in particular at which interface and to what extent data is exchanged for data transfer, data processing, data storage and / or data generation, then analyzes the detected data amounts by means of an analysis unit which has a suitable and

[0015] 3

[0016] The processor is set up and assigns the data movement to a server-side logic, functionality, interface, storage location, fault and / or a predefined error, and transmits this assignment to the front end so that it can be displayed there by means of the display device on the graphical user interface (GUI).

[0017] The monitoring system includes, in particular, a test unit, an analysis unit and a display device.

[0018] The test unit includes, for example, components, modules, and / or devices for measuring data volumes in data storage, RAM, and / or data transfer rates in neural network systems for LLMs and chatbots, as well as in associated processors, RAM, computer networks, and information systems. It also includes components for measuring or determining the current storage capacities of hard drives, such as HDDs and SSDs, and other storage media with capacities in bytes, GB, or TB. This is done, for example, with sensors that measure data transfer in units such as bytes per second, as a performance or energy quantity per unit of time.

[0019] The monitoring system includes, for example, a CPU (Central Processing Unit) that measures computing power in millions or billions of calculations per second. To do this, the CPU accesses the system's or a system component's main memory (RAM) to read and write data. The CPU then transmits the data from the AI-based components of the system to the monitoring system's analysis unit.

[0020] The test unit measures the data transfer at one or more interfaces of the system, for example, at the interface between the back end and the front end of the communication device. In particular, the monitoring system, using the test unit, monitors the incoming and outgoing data traffic at network interfaces, including data transfer rates, number of packets, and / or total data volume.

[0021] The results from the test unit are forwarded to the analysis unit. The analysis unit includes at least one processor configured and aligned to assign the data transfer results to specific faults and / or applications. For example, the following faults are assigned: 202505839

[0022] 4

[0023] RAM and data transfer lines overloaded

[0024] IT security concept threatened

[0025] System firewall breached or attempted breach; detection of malware, phishing and / or ransomware

[0026] CPU usage too high for further input

[0027] Network quality too poor for data transfer to IoT

[0028] Analysis of the inputs does not yield a clear task.

[0029] For example, the following applications are assigned:

[0030] Speech recognition, recording of voice input by the user and / or “active listening”,

[0031] Text processing, recognition and analysis of user text input, data processing, calculation, iteration loops to generate a result, video analysis, facial expression, speech, and gesture recognition.

[0032] Processing of the analyzed inputs by neural networks, possibly including data querying from the IoT.

[0033] Data cleansing, removal of unnecessary information,

[0034] Results of an AI-based calculation in response to a user request, assessment of the reliability of the response, e.g. in percent,

[0035] - Consideration of a proposed solution,

[0036] Assessment of the risks of performing an operation, e.g. in percent.

[0037] The analysis unit provides the result of this mapping to the monitoring system. The monitoring system then generates, for example, an output for the front end, which can be displayed visually and / or audibly.

[0038] The term "front-end" refers to the part of an application or system that is responsible for direct interaction with the user.

[0039] More precisely, the front end includes:

[0040] 1. At least one graphical user interface (GUI) with display and control devices such as:

[0041] Monitor, tablet, display with or without touchscreen functionality, at least partially, microphone, speaker, camera and / or a display device such as a monitor and / or display with visual elements such as an avatar, the latter 202505839

[0042] 5

[0043] This can be used both as a profile picture and as a virtual assistant, including windows, menus, form input masks, and the design and / or layout of the interface.

[0044] 2. For user input and / or interaction with the user:

[0045] o Capture of user input via keyboard, mouse, touchscreen, etc. o Processing of user interactions such as clicks, scrolls, inputs, and appropriate transformation and / or forwarding.

[0046] 3. Presentation of content and data:

[0047] o The rendering and display of text, images, graphics, and multimedia elements.

[0048] o The formatting and preparation of the content for user-friendliness.

[0049] 4. User-side logic and / or functionality:

[0050] o Implementation of business logic and application functions in the browser, o Interactions with the back-end via APIs.

[0051] The front end thus forms the interface between humans and machines and is responsible for the visual presentation and usability of an application. It is typically implemented using technologies such as HTML, CSS, and JavaScript.

[0052] The term "back-end" refers to the part of an application or system that is located behind the user interface and is responsible for the server-side logic and functionality.

[0053] The back end typically includes the following components:

[0054] 1. Server infrastructure:

[0055] o Web server, application server, database server

[0056] o Operating systems, virtual machines, containers

[0057] 2. Server-based application logic:

[0058] o Business logic and calculations

[0059] o Data management and processing

[0060] 3. Data storage:

[0061] o Databases (SQL, NoSQL)

[0062] o File systems and storage solutions

[0063] 4. Interfaces and APIs: 202505839

[0064] 6

[0065] o Provision of programming interfaces

[0066] o Communication with the front end

[0067] 5. System integration:

[0068] o Integration of external systems and services

[0069] o Data flows and synchronization

[0070] 6. Security and Administration:

[0071] o Authentication, authorization, access control

[0072] o Monitoring, logging, backup systems

[0073] The back end is therefore responsible for server-side functionality, data processing, and system integration. It forms the "backbone" of an application and is typically implemented using technologies such as Java, .NET, PHP, Python, and Node.js.

[0074] “Programmable Logic Controller” means that the communication device, with its front end, relates to the back ends of systems and / or devices whose functionality is controlled by a stored program. Typical programmable logic controllers (PLCs) with such communication devices include: industrial controllers, microprocessor-based devices such as household appliances, mobile phones, embedded systems such as vehicle control units, industrial robots, voice, gesture, and / or text assistants for control systems, voice, gesture, and / or text assistants for robotics, and computer systems in general.

[0075] For example, AI-based visual output devices, such as avatars, which can be used as part of the front end of such programmable logic controllers (PLCs), currently lack visualizations indicating the state of the AI ​​system. However, feedback regarding the various system states is advantageous, if not essential, for the user to interact efficiently and / or correctly (i.e., unambiguously) with the system via the communication device.

[0076] A visual representation of the system states of the AI ​​may be necessary, for example, to prevent the system from being overloaded with information in one area and / or receiving too little or too much time to process user input in another. Or it may be necessary for the user to recognize that the system needs further information and input to generate meaningful results. On the other hand, it is also important for 202505839

[0077] 7

[0078] The user needs to be informed when the computer has gone astray, i.e., is unable to generate reliable or meaningful results. Therefore, monitoring and visual output of the back-end status, especially the back-end processing results, by a suitable monitoring unit is important, as this provides the user with this information.

[0079] Examples of system states of AI systems that can be represented in this way are, for example, "active listening", "processing data", "retrieving data from other sources".

[0080] It is also possible, for example, to depict situations where the monitoring system detects that the AI ​​is not responding based on empirical knowledge, i.e., training, pattern, and / or data, but rather responding generically, i.e., creatively. This is particularly relevant if the monitoring system retrieves and / or records the AI-based assessment of the reliability of the provided result. In other words, the monitoring system could, for instance, make the avatar blush if its own AI-based assessment of the answer's reliability falls below a certain threshold. This allows the monitoring system to indicate when the avatar is "lying."

[0081] The monitoring system displays what it detects in the current back-end situation of the communication device, classifies it, and forwards the detection to the front-end and / or makes a corresponding output.

[0082] For example, in the event of an acute cybersecurity threat, the monitoring system triggers a red warning signal along with a beeping sound on the front end.

[0083] For example, when an interactive system uses voice output, the monitoring system displays an avatar with a visually represented, virtually modifiable brooch and / or a modifiable button located on the avatar's lapel, featuring a simple graphic design.

[0084] For example, it could be a button, such as a round button, with various symbols in colors and / or gradients that appear and disappear from the inside.

[0085] 8

[0086] It outputs outward-moving and / or inversely circular motion representations and / or circumferential circular motion representations. Such a button can, for example, be flat or semicircular.

[0087] Figure 1 shows a schematic structure of a communication device with a monitoring system according to an exemplary embodiment of the invention.

[0088] The back end 1 and front end 10 of the communication device are visible, each with various components 2 to 7 and 11 to 16, as well as the monitoring system 17, shown here as an example interposed unit between these two units 1 and 10. The back end 1 shows the processor unit 2, several processors 3 and 4, some of which are AI-based, and connections to internal and external servers 5, as well as to the IoT, represented here as cloud 6.

[0089] The back end 1 is connected to the front end 10 of the communication device via lines 8. The data volumes exchanged via these lines are recorded and analyzed by the monitoring system 17. According to the exemplary embodiment of the invention shown here, the monitoring system 17 is connected to all interfaces – represented by double arrows – of the communication device, so that it can record, process, and assign all information about data processing within the communication device.

[0090] Between the front-end 10 with the display devices 12, 13 and 14 on whose graphical user interfaces (GUI) the assignment of the results of the monitoring system can be displayed and the back-end 1 with the server 5, the unit with the neural network 2, 3 and 4 and the access to the cloud 6 and / or the IoT (Internet of Things) there is an interface with data transfer 8, which provides the monitoring unit 17 with information about which and how much data is currently being transferred.

[0091] While the back end handles data processing and results generation, the front end involves input, receiving results, and output and / or display for the human user. In the embodiment shown here, for example, the monitoring system's mapping of data movements within the communication device is included in the display. The output is displayed, for example, on a monitor 13 with a screen or display showing an avatar 14, a virtual personality. This avatar can make a statement, gesture, or facial expression, such as looking into a [202505839]

[0092] 9

[0093] looking at a specific corner, putting his hand behind his ears to indicate that he wants to hear something, meaning further input from the user is required to process the information, or something similar.

[0094] The front end, as shown here, includes in addition to the screen with the avatar display a computer unit 11, which is suitable and configured for image processing or for generating visual and / or acoustic data from the data received by the monitoring system 17.

[0095] The visual and / or acoustic data from computer unit 11 are forwarded to output devices 12 and 13. The results of the monitoring system are also forwarded to output devices 12 and 13 so that the user can receive (not display) the information there. The user receives it because they listen and / or look.

[0096] In one exemplary embodiment of Avatar 14, it is represented by a button on the lapel.

[0097] This is illustrated, for example, in Figures 2a, 2b and 2c.

[0098] Figure 2a shows a female avatar with a flat, round button 20. The inner circle of the button is illuminated, while a surrounding, but only partially illuminated, segment of the circle is indicated by an outer circle encircling the button, which is illuminated along that segment. This button 20 indicates that data is currently being processed in the back end. In addition to this symbolic indication with the illuminated button, the avatar's head posture, gestures, and facial expressions also suggest that she is "thinking"—the communication device is currently calculating a response to the user's input and preparing for a future output of a response or result.This is indicated firstly by the button that can be seen on the monitor of the output device of the communication device, and this indication is further emphasized by the head posture and facial expressions / gestures and the thoughtful expression of the avatar, who remains silent during this period.

[0099] Figure 2b again shows the same female avatar as Figure 2a; this time the communication device indicates listening, which is again clearly shown by the brightly lit button. 202505839

[0100] 10

[0101] Recognizable are, firstly, the fully illuminated inner circle and, secondly, the fully illuminated surrounding outer circle. Here, for example, the communication device is in a listening state, particularly represented by the avatar, which in this sequence displays all the characteristics of active listening, such as full attention, eye contact, and an engaged posture.

[0102] Figure 2c again shows the female avatar, this time the button is completely unlit because the communication device itself is making a statement and speaking and / or answering at the moment shown here, represented here by the avatar speaking, output via loudspeaker.

[0103] In one exemplary embodiment of the invention, the button flashes when an error and / or malfunction is displayed.

[0104] Naturally, all possible switching configurations of the button, representing various states of the communication device according to an exemplary embodiment, can be derived from the illuminations of the button and the surrounding circular line shown here in Figures 2a to 2c and are included in the scope of the invention.

[0105] The invention has been described with reference to exemplary embodiments. However, variations and modifications are also included within the scope of the invention as presented by the claims, the description, and the figures. The foregoing description is therefore to be understood as exemplary and not as limiting, and all equivalent and / or combinations of embodiments are included in this description.

[0106] Regardless of the grammatical gender of a particular term, persons with male, female or other gender identities are included.

[0107] The invention provides a monitoring system with visual output that, for the first time, informs the user of the status of the communication device simply, intuitively, and through clear symbols. In particular, the user understands whether a malfunction exists, whether the communication device is currently processing data, waiting for input, and more. For example, the user receives visual feedback from the system that allows them to assess which actions are currently appropriate and which are not.

Claims

202505839 11 Patent claims 1. Monitoring system with a display device for a programmable logic controller (PLC) communication device, wherein the display device is part of a device suitable for user interaction and comprises at least a graphical user interface (GUI) and one or more interfaces (APIs) with the back end of the communication device at the front end, wherein the monitoring system has a monitoring unit with which it records the data flow and / or the amount of data exchanged between the back end and the front end, in particular at which interface and to what extent data is exchanged for data transfer, data processing, data storage and / or data generation, then analyzes the recorded data volumes by means of an analysis unit comprising a suitable and configured processor that assigns the data movement to a server-side logic, functionality, interface, storage location, fault and / or a predefined error.This assignment is transmitted to the front end so that it can be displayed there on the graphical user interface (GUI) using the display device.

2. Monitoring system according to claim 1, comprising a testing unit, an analysis unit and a display device.

3. Monitoring system according to claim 1 or 2, wherein the test unit measures the incoming and outgoing data traffic at the interface between the front-end and back-end of the communication device.

4. Monitoring system according to one of the preceding claims, comprising an analysis unit.

5. Monitoring system according to one of the preceding claims, wherein the analysis unit performs an assignment of the results of the testing unit.

6. Monitoring system according to one of the preceding claims, wherein the output of the assignment results by the analysis unit is visual. 202505839 12 7. Monitoring system according to one of the preceding claims, wherein the output of the assignment results by the analysis unit is acoustic.

8. Monitoring system according to one of the preceding claims, wherein the output of the results of the assignment by the analysis unit is visualized by an avatar.

9. Monitoring system according to one of the preceding claims, wherein the output of the results of the assignment by the analysis unit is visualized by an avatar.

10. Monitoring system according to one of the preceding claims, wherein the output of the results of the assignment by the analysis unit is visual via a button.

11. Monitoring system according to one of the preceding claims, wherein the output of the results of the assignment by the analysis unit is visually provided by a button that lights up.

12. Monitoring system according to one of the preceding claims, wherein the output of the results of the assignment by the analysis unit is visually provided by a button that flashes.

13. Monitoring system according to one of the preceding claims, wherein the output of the results of the assignment by the analysis unit is visual and is provided by a button which, depending on the result of the assignment, lights up and / or flashes in different colors.

14. Monitoring system according to one of the preceding claims, wherein the output of the assignment results by the analysis unit is visualized by means of a button with an outer circumferential circle. 202505839 13 15. Monitoring system according to one of the preceding claims, wherein the output of the results of the assignment by the analysis unit is visual via a button with an outer circumferential circle that lights up and / or flashes section by section.

Images