Low‑resource real‑time multi‑screen synchronization and adaptive interaction display framework

Abstract

The present invention provides a low‑resource real‑time multi‑screen synchronization and adaptive interaction display framework. Display is driven centrally; the present invention has low requirements for the display capability of a display control terminal and low requirements for a terminal processor and memory, greatly reduces device costs, is suitable for use in embedded devices, supports heterogeneous terminals with different resolutions, does not need application customization and modification for devices, and ensures the consistency of display content; an interpolation adaptive algorithm ensures clear display after content scaling; the present invention is compatible with various screen sizes, and has low software and hardware maintenance costs.

Description

A Low-Resource Real-Time Multi-Screen Synchronization and Adaptive Interactive Display Framework Technical Field

[0001] This invention relates to the field of embedded device display and interaction technology, specifically to a low-resource real-time multi-screen synchronization and adaptive interactive display framework. Background Technology

[0002] In embedded device environments, memory and processing power are often limited. With the increasing number of display devices and the improvement in resolution, the demand for multi-screen synchronization is growing. This is especially true in scenarios such as education and training, command and control, and industrial site monitoring, where the diversity of devices (large resolution differences) further increases complexity. This makes an efficient and lightweight synchronous display framework urgently needed. In embedded systems, screen display and human-computer interaction are key components. However, existing technologies face challenges in terms of resource consumption, real-time performance, and scalability. Current mainstream technology frameworks can be divided into the following categories:

[0003] 1. Point-to-point transmission technology:

[0004] Point-to-point transmission is the most common method of terminal communication, exchanging data through a dedicated connection between the server and the terminal. The advantage of this method is its simple architecture and ease of implementation. However, as the number of terminals increases, the server needs to maintain a large number of connections, leading to a sharp increase in bandwidth consumption and computing resource demands, making it difficult to meet the needs of multi-terminal synchronization. This can especially impact real-time performance and overall system performance in large-scale display or multi-screen systems.

[0005] 2. Broadcast distribution technology:

[0006] Broadcast technology achieves synchronized display by sending data to all devices in a network at once. This method is efficient in a single network scenario, but the lack of selectivity in broadcast data leads to significant bandwidth waste. Furthermore, broadcast technology places high demands on the receiving and parsing capabilities of the terminal, making it unsuitable for resource-constrained embedded devices.

[0007] 3. Terminal graphics solutions and solutions based on high-resource devices:

[0008] Some technical solutions run independent graphics applications on display terminals, relying on high-performance servers and terminal devices to achieve multi-screen display synchronization. These solutions typically employ complex graphics rendering and transmission algorithms, such as high-definition streaming media encoding and decoding, which place high demands on the resource capabilities of the display terminals and involve complex technologies, such as requiring the running of graphics frameworks, transmission of large amounts of information, local processing, and synchronization with other display terminals. However, this high resource dependence makes it difficult to promote the application of such solutions in low-cost, low-resource embedded environments.

[0009] Therefore, current technology has significant shortcomings in the following aspects:

[0010] 1. High resource requirements: Existing technology frameworks place high demands on the processor, memory, display capabilities, and network bandwidth of devices, which embedded devices struggle to meet. In particular, display terminals running graphics frameworks incur high synchronization costs and resource requirements, making high-resource architecture solutions unsuitable for embedded systems with limited memory and insufficient processor performance.

[0011] 2. Poor real-time performance: Point-to-point transmission schemes suffer from high latency in multi-terminal scenarios, hindering rapid response. Scalability is limited. Performance degrades rapidly in multi-terminal environments.

[0012] 3. Insufficient adaptability: Traditional frameworks cannot provide adaptive display for different screen resolutions, and cannot adapt to various heterogeneous display terminals and resolutions, resulting in inconsistent user experience and poor scalability.

[0013] 4. Complex Synchronous Interaction: Multi-terminal display and control devices need to synchronously display graphics devices and respond to control commands, and update the displayed graphics. The technology and process are complex, and the accuracy of display data synchronization is difficult to guarantee. Especially under network jitter, the quality of the asynchronous display of the graphical interface is poor. Summary of the Invention

[0014] The technical problem that the present invention needs to solve is:

[0015] 1. How to achieve synchronous display and real-time interaction of multiple screen terminals with low resource consumption when embedded devices have limited resources?

[0016] 2. How to reduce bandwidth usage and server computational overhead through efficient data transmission protocols (such as multicast) and synchronization frameworks? How to reduce bandwidth usage while meeting the real-time synchronization requirements of multiple screens?

[0017] 3. How to adapt to screens of different sizes and resolutions, adaptively display graphics, and ensure clear graphic content and consistent layout.

[0018] 4. How to improve the real-time performance of interaction, reduce response latency in the human-computer interaction process, and increase the heterogeneity and scalability of display terminals.

[0019] The technical solution of this invention provides a low-resource real-time multi-screen synchronization and adaptive interactive display framework, including a display control terminal, a central server module, and a communication transmission module. The display control terminal and the central server module establish a control multicast group and a graphics multicast group through the communication transmission module.

[0020] The display and control terminal in use captures user input events and screen coordinate position requests, assembles them into a communication protocol format, and sends them to the communication transmission module along the control multicast group.

[0021] The communication transmission module uses UDP multicast to parse user input events and screen coordinate position requests into data and control commands, which are then transmitted to the central server module along the control multicast group.

[0022] The central server module acquires data and control commands, and uses the communication transmission module to manage the legitimacy of multiple display and control terminals. For the verified display and control terminals in use, the graphics framework runs to perform graphical interface and rendering according to the data and control commands, and responds to generate new graphics. According to the graphics display strategy, the graphics content is read and encapsulated into graphics data in the video memory, and the graphics data in the video memory is sent to the communication transmission module along the graphics multicast group according to the protocol rules.

[0023] The communication transmission module uses UDP multicast to transmit the graphics data in the video memory along the graphics multicast group to the display and control terminal corresponding to the control command;

[0024] The display control terminal that receives the graphics data from the video memory verifies and parses the graphics data, adapts the parsed graphics data to screens of different resolutions to achieve adaptive graphics, and then flashes it into the video memory for display.

[0025] Preferably, the display control terminal verifies the central server module, and if the verification is successful, it receives the graphics data in the display memory transmitted by the central server module through the communication transmission module.

[0026] Preferably, the central server reads the video memory address mapped by the video memory graphics data in segments, encapsulates and sends the segments to the display and control terminal corresponding to the control command through a multicast multipoint communication protocol.

[0027] Preferably, the legality management step includes:

[0028] Establish multicast groups for device management across multiple display and control terminals;

[0029] The central server joins the device management multicast group and runs the receive task;

[0030] When the display and control terminal is powered on, it announces that it has joined the device management multicast group and sends a key string to the central server through the communication transmission module. After the central server authenticates the declared password, it includes the legitimate display and control device in the management and only accepts control commands from the legitimate display and control device.

[0031] Preferably, the graphic size is dynamically adjusted by interpolation based on the data packet offset and the local resolution to stretch the graphic, ensuring display efficiency and quality, so as to adapt the parsed video memory graphic data to screens of different resolutions.

[0032] This invention proposes a low-resource, real-time multi-screen synchronization and adaptive interactive display framework. It adopts a centrally driven display, which has low requirements for the display capabilities of the display control terminal and low requirements for the terminal processor and memory, greatly reducing equipment costs. It is suitable for embedded devices and supports heterogeneous terminals with different resolutions. It does not require customized application modifications for specific devices, ensuring consistent display content. The interpolation adaptive algorithm ensures that the content remains clear even after scaling. It is compatible with various screen sizes and has low software and hardware maintenance costs. Attached Figure Description

[0033] Figure 1 is a schematic diagram of the centrally driven synchronous display in a low-resource real-time multi-screen synchronization and adaptive interactive display framework provided by an embodiment of the present invention. Detailed Implementation

[0034] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0035] This invention provides a low-resource real-time multi-screen synchronization and adaptive interactive display framework, involving low-resource configuration requirements of display and control terminals, plug-and-play discovery mechanisms, centrally driven multi-screen synchronization and interaction technology, and adaptive graphics resolution display capabilities, including:

[0036] 1. Central Server Module

[0037] The central server is the core of the entire system, responsible for graphics generation, data distribution, and event handling. It includes the following sub-modules:

[0038] 1) Terminal Management: Legality management of the discovery, joining, authentication, and leaving processes of display and control terminals, enabling plug-and-play flexible expansion.

[0039] 2) Graphics Generator: Runs the graphics framework, implements the human-computer interaction interface and content, and is responsible for generating and rendering the graphics interface according to the default resolution.

[0040] 3) Virtual device driver: Implement virtual keyboard and mouse drivers, receive human-computer interaction events (mouse, keyboard, touch, etc.) uploaded by the display and control terminal, respond to and process them to generate new graphics.

[0041] 4) Graphics data distribution module: According to the graphics display strategy, reads and encapsulates the graphics content, and sends the graphics data in the video memory according to the protocol rules.

[0042] 5) Control command distribution module: Reads human-machine interaction control commands from display and control terminal devices that meet the identity authentication criteria.

[0043] 2. Display and control terminal

[0044] The display and control terminal is responsible for receiving and displaying graphic data, while also capturing user operation events and sending them back to the central server. It includes the following main sub-modules:

[0045] 1) Device Management Module: Declare discovery protocol and display device information, and perform authentication.

[0046] 2) Data Receiver: The receiving server verifies and parses the graphic data packets sent by multicast.

[0047] 3) Graphics adapter: Adapts the parsed graphics data to screens of different resolutions to achieve adaptive graphics and flashes it into the video memory for display.

[0048] 4) Event capture module: Captures user input events (such as touch, keyboard, mouse) and screen coordinate position requests, assembles them into a communication protocol format, and sends them to the server over the network.

[0049] 3. Communication transmission module

[0050] It employs UDP multicast for data and control command transmission, supporting efficient one-to-many data distribution and command feedback, reducing bandwidth overhead from repeated transmissions, and ensuring the reliability and order of event transmission. It includes the following submodules:

[0051] 1) Achieve efficient distribution of graphical display data based on multicast groups and UDP communication protocols.

[0052] 2) Implement plug-and-play protocol distribution for devices based on multicast groups and UDP communication protocols.

[0053] 3) Implement human-computer interaction event communication based on multicast group and UDP communication protocols.

[0054] Figure 1 shows a schematic diagram of the central drive synchronous display.

[0055] The embodiments of the present invention include the following technical implementations:

[0056] 1. Center-driven graphics generation and multi-screen display

[0057] The centrally driven multi-screen display synchronization technology mentioned in this invention differs from traditional local graphics display and network-based distributed display overlay technologies. The key technology lies in the fact that the graphics application software runs entirely on the central server, generated entirely by the central server; the display control terminal does not run any graphics generation software locally. The central server runs graphics frameworks such as QT and GTK. The GPU renders at a default resolution, such as 1024*768, and generates the graphical interface and display content according to the user-designed application. The rendered graphical interface is then stored in the central server's video memory and made available for subsequent applications to read through a video memory mapping interface. The graphics distribution application on the central server reads the video memory address mapped to the generated image in segments via a multicast multipoint communication protocol. The read graphics information data is encapsulated according to the protocol and sent to the display control terminal registered in the multicast group. The display terminal does not need to run a graphics framework; it only needs to scale and adapt the received image to its local resolution according to an algorithm and write it to its local video memory for display. This greatly reduces the resource requirements and equipment costs of the display terminal, including memory, graphics card, and computing power. Graphical interface upgrades can be performed solely on the central server, significantly reducing upgrade and maintenance costs.

[0058] 2. Plug and play discovery and registration of display and control terminals

[0059] For security reasons, a security discovery and control mechanism is designed between the multi-display control terminals and the central server. Registration and management of the display control terminals are achieved through a multicast group managed by the device management system. The central server joins the device management multicast group and runs a receiving task. When a display control terminal powers on, it announces its joining of the device management multicast group and sends a key string to the central server. The central server authenticates the terminal based on the announced password and then adds the legitimate display control device to its management. Thus, the display control terminals can be used plug-and-play without any configuration parameters.

[0060] 3. Human-computer interaction technology

[0061] The display / control terminal typically connects to a keyboard, mouse, or touch device to achieve graphical human-computer interaction. The focus of human-computer interaction and display driven by the central server lies in the acquisition of data from the display / control terminal and the response to events from the server. The central server integrates and converts remote human-computer interaction events into the local graphics framework through a virtual keyboard and mouse driver. After the display / control terminal device acquires the user's human-computer interaction events, such as keyboard key values, mouse event coordinates, and touch device event coordinates, it encapsulates them into a predefined communication protocol format and sends the events to the control multicast group. The central server runs a control event receiving program, restores the received interaction events, and uses a virtual mouse and keyboard driver to write the restored keyboard and mouse events into the virtual keyboard and mouse driver, which is then sent to the local graphics framework on the central server. The graphics framework responds to the events and position requirements, generates a new graphical interface, refreshes it to the local video memory, and then sends it to the display / control terminal through a sending program.

[0062] 4. Multi-screen low-resource graphics transmission and display technology

[0063] Traditional multi-screen synchronization technology requires significantly more resources as the number of display and control terminals increases, placing higher demands on data communication and the central server. This invention employs multicast technology to transmit graphics and images, enabling real-time communication and display of new graphics among display and control terminals within a multicast group. This minimizes communication resource requirements, achieving resource overhead independent of the number of display and control terminals.

[0064] Since graphics are transmitted via video memory data acquisition, this approach leverages the characteristics of multicast UDP propagation and employs direct video memory mapping. After mapping, graphics data is efficiently read through the driver interface. A video memory segmentation and numbering transmission technique is used, segmenting the video memory into numbers. Each transmission carries graphics information smaller than one Ethernet MTU, effectively preventing large-packet UDP transmissions from being discarded by the network protocol stack due to fragment loss and inability to reassemble. The receiving end receives segments with numbers below the MTU, writes the content to local video memory based on the offset in the data packet, and renders it to the screen via the display driver on the display control terminal. This scheme enables partial image data refresh, ensuring correct display even if some non-critical numbered graphics are lost. Further optimization measures include maintaining a dual buffer in the terminal memory: one for the current display and one for receiving new graphics, preventing screen tearing.

[0065] 5. Resolution Adaptive Technology

[0066] The display and control terminal dynamically adjusts the graphic size based on the data packet offset sent by the server and the local resolution, using algorithms such as interpolation to stretch the graphics and ensure display efficiency and quality. This enables local adaptive display of the human-computer interaction interface and content based on a standard graphic at a fixed resolution (e.g., 1024×768).

[0067] The implementation process of this invention is as follows:

[0068] 1) The central server starts initialization, runs the device management submodule, joins the graphics data communication multicast group and the control interaction command multicast group, runs the authentication module, starts the virtual keyboard and mouse driver, runs the application, and renders and generates graphics.

[0069] 2) The device management submodule of the display and control terminal starts up, joins the default control interaction multicast group, joins the graphics data communication multicast group, runs the local graphics adaptive task and control command acquisition task, allocates memory and maps it to the video memory of the local display device.

[0070] 3) The display and control terminal sends device announcement information to the default control interaction multicast group according to the device discovery protocol format. The information includes the authentication password.

[0071] 4) The central server will authenticate the received announcement protocol, and the IP address of the display and control terminal that passes the authentication will be added to the allowed control list;

[0072] 5) The central server graphics generation submodule renders the human-computer interaction interface designed by the user application into graphics at a standard resolution of 1024*768, and outputs the graphics data to the central server's video memory through mapping technology.

[0073] 6) The graphics data distribution module of the central server reads and numbers the graphics data in the video memory in real time in a loop, with a length of less than one Ethernet IP data packet MTU. The corresponding numbered data is encapsulated according to the designed communication protocol format and sent to the graphics data communication multicast group. When the current round of video memory graphics data is sent, a special frame with a sending offset of 0x80000000 is added as the end frame to notify the display control terminal that the current round of video memory graphics data has been sent.

[0074] 7) The data receiver of the display and control terminal receives the multicast graphics data and copies the graphics data to the mapped memory address in sequence. When a special frame with a transmission offset of 0x80000000 is detected, it is considered that the data transmitted is complete. The graphics adaptive algorithm is executed to scale the standard resolution of 1024*768 graphics data to adapt to the local resolution such as 1920*1080 and render it to the local video memory.

[0075] 8) The display and control terminal shows the latest human-computer interaction graphic data generated by the central server;

[0076] 9) Users operate the display and control terminal via keyboard, mouse, touch, etc. The event capture module of the display and control terminal captures keyboard, mouse, touch, and other events of the display and control device, and encapsulates the event, position coordinates, and other information into the format specified by the control protocol and sends it to the control interaction command multicast group.

[0077] 10) After receiving the control command sent by the display control, the central server control command receiving module verifies the source address of the command, discards the control command sent by the illegal device, and writes the event type and location information of the control command of the legal device into the virtual keyboard and mouse driver of the central server and sends it to the graphics display frame.

[0078] 11) The central server's graphical display framework receives events and position information sent by the virtual keyboard and mouse driver, and generates new human-computer interaction interface graphical information in response.

[0079] 12) The central server proceeds to step 5 to begin the current round of updating graphics memory information. This process repeats until the central server shuts down or stops working.

[0080] 13) Multiple display and control terminals shall perform the joining step in step 3;

[0081] 14) The display and control terminal sends a departure announcement protocol to leave the frame.

[0082] The beneficial effects of the embodiments of the present invention are as follows:

[0083] 1. Low resource consumption: The centrally driven display has low requirements for the display capabilities of the display control terminal and low requirements for the terminal processor and memory, which greatly reduces the cost of the equipment and is suitable for use in embedded devices.

[0084] 2. High-efficiency data transmission: Both graphics and control use multicast protocols, reducing redundant data transmission, simplifying software technology, and saving bandwidth.

[0085] 3. Adaptive display: Centrally driven, it supports heterogeneous terminals with different resolutions, without the need to customize applications for specific devices, ensuring consistent display content. The interpolation adaptive algorithm ensures that the content is still clearly displayed after scaling, and it is compatible with various screen sizes, with low software and hardware maintenance costs.

[0086] 4. Real-time interaction: Centralized management and control of commands and generation of graphics technology, small data transmission volume, avoidance of fragmented retransmission, reduced interaction latency, and improved user experience.

[0087] 5. High scalability: Plug-and-play configuration technology supports concurrent operation on multiple terminals without significantly increasing server load. It can support multiple concurrent terminals with stable server performance and resource consumption.

Claims

1. A low-resource real-time multi-screen synchronization and adaptive interactive display framework, characterized in that, It includes a display and control terminal, a central server module, and a communication transmission module. The display and control terminal and the central server module establish a control multicast group and a graphics multicast group through the communication transmission module. The display and control terminal in use captures user input events and screen coordinate position requests, assembles them into a communication protocol format, and sends them to the communication transmission module along the control multicast group. The communication transmission module uses UDP multicast to parse user input events and screen coordinate position requests into data and control commands, which are then transmitted to the central server module along the control multicast group. The central server module acquires data and control commands, and uses the communication transmission module to manage the legitimacy of multiple display and control terminals. For the verified display and control terminals in use, the graphics framework runs to perform graphical interface and rendering according to the data and control commands, and responds to generate new graphics. According to the graphics display strategy, the graphics content is read and encapsulated into graphics data in the video memory, and the graphics data in the video memory is sent to the communication transmission module along the graphics multicast group according to the protocol rules. The communication transmission module uses UDP multicast to transmit the graphics data in the video memory along the graphics multicast group to the display and control terminal corresponding to the control command; The display control terminal receives the graphics data from the video memory, verifies and parses the data, adapts the parsed graphics data to screens of different resolutions to achieve adaptive graphics, and then flashes it into the video memory for display.

2. The low-resource real-time multi-screen synchronization and adaptive interactive display framework as described in claim 1, characterized in that, The display control terminal verifies the central server module. If the verification is successful, it receives the graphics data in the display memory transmitted by the central server module through the communication transmission module.

3. The low-resource real-time multi-screen synchronization and adaptive interactive display framework as described in claim 1, characterized in that, The central server reads the video memory address mapped by the video memory graphics data in segments, encapsulates and sends them to the display and control terminal corresponding to the control command through multicast multipoint communication protocol.

4. The low-resource real-time multi-screen synchronization and adaptive interactive display framework as described in claim 1, characterized in that, The legality management steps include: Establish multicast groups for device management across multiple display and control terminals; The central server joins the device management multicast group and runs the receive task; When the display and control terminal is powered on, it announces that it has joined the device management multicast group and sends a key string to the central server through the communication transmission module. After the central server authenticates the declared password, it includes the legitimate display and control device in the management.

5. The low-resource real-time multi-screen synchronization and adaptive interactive display framework as described in claim 1, characterized in that, Based on the data packet offset and the local resolution, the graphic size is dynamically adjusted using interpolation to stretch the graphic, ensuring display efficiency and quality, so as to adapt the parsed video memory graphic data to screens of different resolutions.

Images