Conference space perception method and apparatus, and electronic device
By combining lightweight monocular depth estimation and semantic segmentation techniques, optimizing the depth estimation model, and integrating static and dynamic information, the problem of insufficient object and person detection accuracy in complex conference room scenarios is solved, realizing a high-precision digital twin model of the conference space, and improving the interactive experience and intelligent management of the virtual space.
Patent Information
- Application Number
- PCT/CN2025/093839
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Filing Date
- 2025-05-09
- Publication Date
- 2026-11-12
AI Technical Summary
Existing 3D spatial perception technology lacks sufficient accuracy in detecting objects and people in complex conference room scenarios, affecting the quality and reliability of digital twins of conference spaces.
By combining lightweight monocular depth estimation and semantic segmentation to obtain semantic point clouds of static meeting environments, and optimizing the head detection model of depth estimation through temporal fusion, the scale of training data is expanded, dynamic pedestrian information and static meeting scene information are fused to construct a high-precision digital twin model of meeting space.
It achieves high-precision object and personnel detection in complex conference room scenarios, enhances the interactive experience of participants in virtual space and the level of intelligent meeting management, and supports remote collaboration and intelligent interaction.
Smart Images

Figure CN2025093839_12112026_PF_FP_ABST
Abstract
Description
A method, device and electronic equipment for sensing meeting space Technical Field
[0001] This application relates to the field of spatial perception technology, and in particular to a method, device and electronic device for spatial perception of meetings. Background Technology
[0002] With the development of digitalization in meeting spaces, digital twin technology has become an important means to enhance the meeting experience. Digital twins of meeting spaces map real-world meeting scenarios onto a virtual space, enabling the digital reconstruction of the meeting environment and participant behavior, thereby supporting remote collaboration, intelligent meeting management, and space optimization. To achieve this, 3D spatial perception is a core technology, capable of acquiring the 3D structure of the meeting room, the real-time location of participants, and their activity trajectories. However, in complex scenarios, existing technologies still suffer from insufficient accuracy in object and person detection in 3D spatial perception, affecting the quality and reliability of digital twins for meeting spaces.
[0003] Application content
[0004] One objective of this application is to provide a conference space perception method, apparatus, and electronic device to solve the technical problem of how to perform high-precision detection of the three-dimensional spatial information of objects and people in complex scenes.
[0005] To address the aforementioned technical problems, one technical solution adopted in this application is: providing a meeting space perception method, comprising: acquiring a target image corresponding to a meeting scene; performing static scene reconstruction on the meeting scene based on the target image to generate static scene information of the meeting scene, the static scene information including three-dimensional structural information of the meeting scene, contour information of the meeting scene, and geometric information corresponding to objects in the meeting scene; acquiring dynamic person data information corresponding to people in the meeting scene based on the target image, the dynamic person data information including the position information and depth information of the people; and fusing the static scene information and the dynamic person data information to generate a meeting space perception result of the meeting scene.
[0006] This method acquires 3D structural, contour, and object geometric information through static scene reconstruction and combines it with dynamic person data to form a complete spatial perception system, ensuring the comprehensiveness and accuracy of the data. By fusing static scene and dynamic person data, this method can not only identify the spatial structure of the meeting environment but also accurately locate participants and track their activities in real time. This capability is particularly crucial for scenarios such as personnel management, behavior analysis, and seating optimization. By generating the meeting space perception results, the system can dynamically update the meeting scene, ensuring a high degree of synchronization between virtual and real meetings, thereby achieving intelligent interaction. Therefore, by integrating static scene reconstruction and dynamic person data to generate meeting space perception results, this method achieves high-precision positioning, real-time updates, and intelligent analysis. It not only improves the intelligence level of meeting management but also provides a solid foundation for applications such as remote collaboration, intelligent interaction, and security monitoring, promoting the development of meeting scenarios towards intelligence and data-driven approaches.
[0007] In some embodiments, static scene reconstruction of the meeting scene is performed based on the target image to generate static scene information of the meeting scene, including: obtaining a depth map and a segmentation map of the meeting scene based on the target image; generating a 3D point cloud with semantic information based on the depth map and the segmentation map, wherein the 3D point cloud with semantic information is the 3D structural information of the meeting scene; obtaining the contour information of the meeting scene based on the 3D point cloud; extracting the point cloud of objects in the meeting scene based on the semantic information of the 3D point cloud, projecting the point cloud of the objects onto a target plane, and calculating the boundary information of the objects based on the target plane onto which the point cloud of the objects is projected. Here, by combining the depth map and the segmentation map, static scene reconstruction of the meeting scene is successfully achieved, generating rich 3D structural information, contour information, and object geometric information. This process not only provides accurate 3D semantic point clouds for the meeting space but also effectively improves the visualization and interactive performance of the scene by fusing semantic information and object boundary data. By accurately reconstructing the static layout of the meeting space and the geometric features of the objects, users can obtain a high-fidelity meeting environment in a virtual space, helping to enhance the realism and immersion of the meeting experience.
[0008] In some embodiments, obtaining the contour information of a meeting scene based on a 3D point cloud includes: obtaining a point cloud of a target plane based on the 3D point cloud, and calculating the minimum convex polygon corresponding to the point cloud projected onto the target plane; connecting the vertices of the minimum convex polygon corresponding to the point cloud to generate the contour information of the meeting scene. Specifically, by using the convex hull algorithm to calculate the minimum convex polygon of the point cloud on the top-view plane and generate the contour, not only can the spatial boundaries of objects in the meeting scene be accurately described, but also reliable data support can be provided for further virtual meeting space modeling. This method can effectively handle complex 3D scenes, ensuring that the spatial layout of the meeting space is realistically and accurately digitally represented, enabling the digital twin system to more realistically reconstruct the meeting scene and enhance the user's immersive experience in the virtual space.
[0009] In some embodiments, dynamic person data information corresponding to persons in a meeting scene is obtained based on a target image. This dynamic person data information includes the person's position and depth information, including: detecting head bounding boxes based on the target image, whereby the head bounding boxes are used to identify the head region corresponding to the person in the meeting scene; obtaining the width and height of the head bounding boxes; inputting the width and height of the head bounding boxes into a depth calculation model to obtain the depth information of the person in the meeting scene; and performing back-projection calculation based on the depth information and the center point coordinates of the head bounding boxes to obtain 3D point cloud data of the person in the meeting scene. This 3D point cloud data constitutes the person's position information. By combining the target image, the depth calculation model, and the point cloud data, high-precision person detection and 3D localization can be achieved. This solution has higher depth estimation accuracy, effectively reduces errors, and ensures accurate fusion of spatial data. It not only enhances the meeting scene perception capability but also provides strong support for applications such as digital twin modeling, remote conferencing interaction, and behavior analysis, enabling intelligent meeting systems to possess a higher level of perception and interaction capabilities.
[0010] In some embodiments, the method further includes: obtaining a depth calculation model; obtaining the depth calculation model includes: detecting a head bounding box based on the target image, the head bounding box being used to identify the head region corresponding to a person in the meeting scene; obtaining the width and height of the head bounding box; obtaining 3D point cloud data corresponding to the meeting scene using LiDAR; transforming the 3D point cloud data to the camera coordinate system corresponding to the target image to generate a camera coordinate system point cloud; mapping the camera coordinate system point cloud to the image coordinate system to obtain a sparse depth map of the meeting scene; obtaining pixels with depth values based on the head bounding box and the sparse depth map, and selecting the minimum depth value as the depth of the head bounding box; calculating target parameters based on the depth of the head bounding box, as well as the width and height of the head bounding box; and determining the depth calculation model based on the calculated target parameters. Specifically, by fusing image detection and LiDAR point cloud data, high-precision person depth estimation is achieved; sparse depth maps are generated using camera coordinate transformation and image projection to ensure accurate matching of depth information with the target image; simultaneously, the target parameters are calculated using the width, height, and depth data of the head bounding box to obtain the depth calculation model, making it adaptable to different scenarios and enhancing its versatility and robustness.
[0011] In some embodiments, based on the head bounding box and a sparse depth map, pixels with depth values are obtained, and the minimum depth value is selected as the depth of the head bounding box. This includes: traversing the region of the head bounding box and extracting the coordinates of the pixels within the head bounding box; searching for the corresponding depth value of each pixel in the sparse depth map based on its coordinates, and filtering to obtain pixels with valid depth values; obtaining the minimum depth value of the pixels based on the valid depth values, and using the minimum depth value as the depth of the head bounding box. The minimum depth filtering strategy effectively removes background interference, improves ranging accuracy, and enhances the robustness of depth estimation to a certain extent, enabling the system to stably provide accurate human depth information under different lighting, occlusion, or complex environments. It also improves the accuracy and stability of human depth perception, providing reliable foundational support for spatial understanding and interactive applications in intelligent conferencing systems.
[0012] In some embodiments, after inputting the width and height of the head frame into the depth calculation model to obtain the depth information of the person in the meeting scene, the method further includes: based on a set sliding window, continuing to acquire a preset number of depth information, each preset number of depth information being obtained through the depth calculation model; calculating the median or average of the depth information obtained within the sliding window; and updating the depth information of the person in the meeting scene based on the median or average. Here, temporal fusion makes the person depth estimation more stable, reducing fluctuations caused by single-frame detection errors, thereby improving the robustness of ranging; it optimizes the person depth perception in the meeting scene, ensuring high-precision and high-reliability depth information in dynamic environments, providing strong support for behavior analysis, intelligent interaction, and visual understanding.
[0013] In some embodiments, static scene information and dynamic person data information are fused to generate a meeting space perception result for the meeting scene. This includes fusing dynamic person data information with the three-dimensional structural information of the meeting scene, the contour information of the meeting scene, and the geometric information corresponding to objects in the meeting scene to generate a meeting space perception result. The meeting space perception result includes the real-time position information and real-time activity information of people in the meeting scene, as well as the environmental features of the meeting scene. Specifically, by effectively fusing static scene information and dynamic person data information, a meeting space perception result is generated. This perception result can reflect the position, activity status, and environmental features of people in the meeting scene in real time, providing accurate data support for meeting management, behavior analysis, space optimization, and interactive design. It not only improves the efficiency of space utilization during meetings but also supports application scenarios such as intelligent interaction and remote participant analysis. Through real-time spatial data feedback, it can greatly enhance the organizational efficiency, participation, and interactivity of meetings.
[0014] To address the aforementioned technical problems, one technical solution adopted in this application is as follows: a meeting space perception device is provided, comprising: an image acquisition module for acquiring a target image corresponding to a meeting scene; a static scene reconstruction module for reconstructing the meeting scene based on the target image, generating static scene information of the meeting scene, the static scene information including three-dimensional structural information of the meeting scene, contour information of the meeting scene, and geometric information corresponding to objects in the meeting scene; a dynamic person data information acquisition module for acquiring dynamic person data information corresponding to persons in the meeting scene based on the target image, the dynamic person data information including the position information and depth information of the persons; and a perception fusion module for fusing the static scene information and the dynamic person data information to generate a meeting space perception result of the meeting scene. This meeting space perception device has the beneficial effects corresponding to the aforementioned meeting space perception method.
[0015] To address the aforementioned technical problems, one technical solution adopted in this application is to provide an electronic device, including a memory and a processor. The memory is connected to the processor, and the processor executes one or more computer programs stored in the memory. When the processor executes the one or more computer programs, it enables the electronic device to implement the meeting space awareness method described above. This electronic device possesses the beneficial effects corresponding to the meeting space awareness method applied to electronic devices described above.
[0016] To solve the above-mentioned technical problems, one technical solution adopted in this application is to provide a non-volatile computer-readable storage medium that stores computer-executable instructions. When the computer-executable instructions are executed by an electronic device, the electronic device performs the meeting space awareness method as described above.
[0017] To solve the above-mentioned technical problems, one technical solution adopted in this application is to provide a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions that, when executed by an electronic device, cause the electronic device to perform the meeting space awareness method as described above. Attached Figure Description
[0018] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.
[0019] Figure 1 is a schematic diagram of an application scenario of a meeting space perception method provided in an embodiment of this application;
[0020] Figure 2 is a flowchart of a meeting space perception method provided in an embodiment of this application;
[0021] Figure 3 is a schematic diagram of the structure of a meeting space sensing device provided in an embodiment of this application;
[0022] Figure 4 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.
[0024] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. Moreover, the terms "first," "second," and "third" used in this application do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.
[0025] It should be noted that in the various embodiments described below, there is no necessarily a certain order between the steps. Those skilled in the art can understand from the description of the embodiments of this application that the steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in interchange, etc.
[0026] A meeting space digital twin is a system that uses digital twin technology to digitally map and virtually reconstruct a real meeting space. It can sense and analyze the meeting room environment, layout, and personnel activities in real time, and build a digital model in the virtual space that is synchronized with the real world. The core goal of this technology is to enhance the meeting experience, making remote conferencing, intelligent meeting management, and space optimization more interactive, intelligent, and real-time.
[0027] 3D spatial perception is a computer vision and sensing technology designed to understand, model, and analyze 3D environments to create digital twins of meeting spaces. It acquires geometric information about a space using devices such as cameras, depth sensors, and LiDAR, and utilizes computer vision, deep learning, and point cloud processing technologies to detect, track, and identify objects, people, and structures within that 3D environment. The core goal of 3D spatial perception is to enable computers to understand the 3D world like humans and to be used in applications such as robotics, autonomous driving, AR / VR, and digital twins.
[0028] Currently, existing 3D spatial perception technologies mainly rely on two types of technologies. One is depth estimation-based spatial perception technology, which infers depth information from a single RGB image using a monocular depth estimation model (such as a deep learning network) and relies on large-scale data training to learn depth relationships in different scenes. The other is spatial perception technology that relies on depth sensors (such as depth cameras), using specialized depth sensors (such as RGB-D cameras and ToF sensors) to directly acquire 3D point cloud data. Although existing 3D spatial perception technologies can perceive the 3D information of objects and people in a conference room to a certain extent during the construction of a digital twin of a conference space, the detection accuracy of objects and people in complex conference room scenarios is still insufficient. For example, depth estimation-based spatial perception technology is limited by the diversity of training data scenarios, resulting in poor generalization ability in unfamiliar conference environments and large depth estimation errors. On the other hand, occlusion by heads and bodies can affect depth estimation, leading to inaccurate estimation of the positions of attendees. Furthermore, the high-precision depth cameras and ToF sensors used in depth sensor-based spatial perception technology are expensive. In particular, ToF sensors are limited by depth resolution, making it difficult to accurately detect small object details (such as pens or mobile phones on a table).
[0029] Therefore, in order to effectively address the problem that existing 3D spatial perception technologies are not accurate enough in detecting objects and people in complex conference room scenarios, it is necessary to explore higher-performance and more comprehensive solutions to build high-precision digital twin models of conference spaces.
[0030] Based on this, embodiments of this application provide a meeting space perception method, device, and electronic device. It primarily achieves real-time meeting scene display by combining high-precision detection of dynamic pedestrian information and static meeting scenes, enhancing the interactive experience of participants in the meeting space. Specifically, for static scenes, a lightweight monocular depth estimation model combined with semantic segmentation is used to obtain the semantic point cloud of the static meeting environment, calculating the 3D information of objects such as walls and desktops. This eliminates the need for high-cost depth cameras, enabling high-precision detection of small object details. For dynamic scenes, a head detection model based on temporal fusion and optimized depth estimation is used to reduce jitter caused by head occlusion and pose changes, improving detection stability. Simultaneously, the training data scale is expanded to improve the model's completion capability, especially in the edge regions of the field of view, where depth values are calculated using a fitting curve based on bounding box height, improving the depth estimation accuracy of edge heads. Finally, dynamic pedestrian information and static meeting scene information are fused to construct a more generalized digital twin of the meeting space, enabling real-time meeting scene display in various types of meeting scenarios, including the layout of the meeting room (scene boundaries, desktops) and the positions and trajectories of dynamic pedestrians in the meeting space.
[0031] The meeting space awareness method proposed in this application is described below through specific embodiments.
[0032] First, the application scenarios of the meeting space perception method according to the embodiments of this application will be introduced. Please refer to Figure 1, which is a schematic diagram of an application scenario of the meeting space perception method provided by the embodiments of this application. The meeting room is equipped with a meeting space digital twin system. This system uses three-dimensional spatial perception technology to map the physical space of the real meeting room to the virtual world in real time, constructing a highly realistic three-dimensional digital meeting environment.
[0033] Before the meeting begins, the system uses a static scene spatial perception algorithm and RGB cameras installed in the meeting room to acquire the three-dimensional structural information of the entire meeting room, including the geometric shape, material texture, and position information of fixed objects such as the conference table, chairs, projection screen, microphone, and whiteboard. This data is integrated into a three-dimensional digital model of the meeting room, and semantic segmentation technology is used to assign semantic labels to each element in the meeting room, such as "lectern," "screen," "aisle," and "seat," for subsequent intelligent management and interaction.
[0034] As attendees entered the room, the system activated its dynamic scene spatial perception algorithm, using head detection and ranging models to capture the position and behavior of participants in real time. Whether it was an executive sitting at the conference table or a team member discussing in a corner of the meeting room, the system could accurately track their position, posture, and movement trajectory, and use temporal fusion technology to reduce depth jitter caused by occlusion or changes in lighting. For personnel located at the edge of the camera's field of view, the system automatically performed edge optimization, using fitted curves to estimate their depth information, ensuring that no personnel were missed in the three-dimensional space.
[0035] When the meeting officially begins, remote participants enter the digital twin meeting space through VR devices or regular computer terminals. They see a three-dimensional scene that is completely identical to the real meeting room and can clearly observe all the people in the meeting room and their actions. By integrating static and dynamic scene data, remote participants can not only experience a realistic meeting environment, but also interact with on-site personnel in the virtual space, such as projecting documents, adjusting seats, and even using virtual whiteboards for collaboration.
[0036] The digital twin system for meeting spaces can also record and analyze the meeting process in real time. For example, the system can automatically mark the location of key speakers, track their movements, and transcribe meeting content using speech recognition technology. For remote participants, the system provides a 360° free-view perspective, allowing them to adjust their viewing angle freely and enhancing immersion. Furthermore, after the meeting, the system generates a meeting data analysis report, including participant interaction, speaking time distribution, and key discussion points, providing valuable meeting insights for corporate management.
[0037] It should be noted that the above application scenario is merely an example of the meeting space perception method provided in the embodiments of this application, and can be set according to the actual situation, without limitation.
[0038] Please refer to Figure 2, which is a flowchart of a meeting space awareness method provided in an embodiment of this application. The method includes steps S11 to S14:
[0039] S11. Obtain the target image corresponding to the meeting scene.
[0040] First, the image acquisition equipment is deployed and initialized. This involves placing multiple high-definition RGB cameras in the meeting room. These cameras can be mounted on the ceiling, in corners, or on desktops to ensure maximum coverage of the entire meeting space and minimize blind spots. These cameras need to have wide-angle or fisheye lenses to capture a wider field of view, and high dynamic range (HDR) cameras should be selected to reduce the impact of changes in lighting (such as projection screen or window lighting). Upon system startup, the cameras undergo automatic calibration, including lens distortion correction, color balance adjustment, and white balance settings, to ensure stable and reliable image quality.
[0041] Secondly, real-time acquisition of target images is performed. This means the camera continuously captures image data within the meeting room at a set acquisition frequency (e.g., 30 frames / second or 60 frames / second). To reduce computational burden, the system can dynamically adjust the acquisition resolution. For example, high-resolution (e.g., 4K or 8K) images are used for static scene acquisition to obtain fine environmental details, while lower-resolution (e.g., 1080P) images are used for dynamic person detection to improve computational efficiency. Target images are stored in color image format (RGB) for subsequent depth estimation, semantic segmentation, and person detection.
[0042] Next, image preprocessing and optimization are performed. Understandably, the acquired target image undergoes preprocessing steps such as noise filtering, color enhancement, and edge sharpening to improve image quality. In the case of a multi-camera deployment, camera calibration algorithms (such as Zhang Zhengyou's calibration method) are used to determine the position and viewing angle of each camera, ensuring that images from different perspectives are correctly aligned and fused. Additionally, motion blur removal algorithms (such as optical flow estimation-based deblurring methods) can be used to reduce image blurring caused by camera shake or rapid movement of people, thereby improving the clarity of the target image.
[0043] Finally, the data is stored and transmitted. Understandably, the acquired target images can be stored in real-time on a local server or in the cloud for subsequent depth estimation, semantic segmentation, and 3D modeling. If the conferencing system supports remote access, the target images need to be transmitted to the remote server via a high-bandwidth, low-latency network (such as 5G or Wi-Fi 6) and compressed using efficient encoding formats such as H.265 and AV1 to reduce bandwidth consumption.
[0044] By acquiring, optimizing, and storing high-quality images, we ensure that subsequent steps have reliable input data, providing accurate basic information for the digital twin of the meeting space.
[0045] S12. Reconstruct the meeting scene statically based on the target image to generate static scene information of the meeting scene. The static scene information includes the three-dimensional structure information of the meeting scene, the outline information of the meeting scene, and the geometric information of the objects in the meeting scene.
[0046] Static scene reconstruction refers to extracting static environmental information of the conference room from a target image using computer vision and depth estimation techniques, and converting it into a 3D point cloud or mesh model to form a 3D digital representation of the conference room. In the digital twin of the conference space, the static scene mainly includes objects that do not change over time, such as: conference room walls (spatial boundaries), floor and ceiling, conference table, chairs, projection screen, and doors and windows. These static objects do not move during the meeting, so they can be modeled and updated at a low frequency to reduce computational overhead.
[0047] Among them, the three-dimensional structural information refers to the spatial geometry of the static scene in the conference room, including:
[0048] The three-dimensional coordinates (x, y, z) of the object: Through monocular depth estimation or multi-view triangulation, the target image is converted into three-dimensional point cloud data, which constitutes the basic framework of the meeting scene;
[0049] Surface morphology: Reconstruct the mesh surface from point clouds to determine the shape of objects, such as the plane of a conference table, the vertical surface of a wall, etc.
[0050] Relative positions between objects: Modeling spatial relationships ensures the model conforms to real-world physical structures. For example, walls should be perpendicular to the ground, and tables should be above the ground.
[0051] Contour information refers to the boundary and shape features of objects in a static scene, primarily used to define the visible range of objects. The functions of contour information include: determining the outline of objects such as conference tables, chairs, and projection screens; using it for subsequent segmentation to help distinguish different objects; and extracting boundary information from images using edge detection algorithms (such as Canny and Sobel) and combining it with depth data to optimize 3D models. For example, in a conference room scene: the outline of a conference table is usually rectangular, the outline of a projection screen may be a large planar area, and the outlines of doors and windows can be used to distinguish entrances from enclosed areas.
[0052] The geometric information corresponding to an object includes its static dimensions, shape, and spatial structure, typically including: Dimensional information: The object's 3D bounding box is calculated using depth data. For example, a conference table might be 2 meters long, 1 meter wide, and 0.75 meters high. Geometric shape: The object's form is determined through point cloud and mesh reconstruction. For example, walls are vertical planes, tabletops are horizontal planes, and some chairs might be curved (like sofas). Topological structure: The connections between different components of the object. For example, the tabletop is connected to the legs, forming a stable structure; the projection screen is fixed to the wall and does not change over time. Material information: In advanced 3D modeling, it may be necessary to record the object's material properties. For example, is the conference table made of wood or glass? Is the projection screen matte or reflective?
[0053] In some embodiments, static scene reconstruction of the conference scene is performed based on the target image to generate static scene information of the conference scene, including: obtaining a depth map and a segmentation map of the conference scene based on the target image; generating a 3D point cloud with semantic information based on the depth map and the segmentation map, wherein the 3D point cloud with semantic information is the 3D structural information of the conference scene; obtaining the contour information of the conference scene based on the 3D point cloud; extracting the point cloud of objects in the conference scene based on the semantic information of the 3D point cloud, projecting the point cloud of the objects onto a target plane, and calculating the boundary information of the objects based on the target plane projected by the point cloud of the objects, wherein the boundary information constitutes the geometric information corresponding to the objects in the conference scene.
[0054] First, depth maps and segmentation maps of the meeting scene are extracted from the target image. For the depth map, a large depth estimation model (such as MiDaS) is used to predict the depth of the target image, i.e., the distance information from each pixel to the camera. Furthermore, since the large depth estimation model is computationally expensive, a lightweight version is trained using distillation learning to make it suitable for low-computing devices such as conference tablets. For the segmentation map, a segmentation model (such as DeepLab) is used to classify the image into different categories, such as "walls," "conference tables," and "chairs." The segmentation map is output, where each pixel corresponds to a category label, used for subsequent semantic point cloud generation. In addition, the depth map and segmentation map are aligned so that the same pixel contains both depth and semantic category information; this aligned data is used for the next step of 3D point cloud generation.
[0055] Next, the obtained two-dimensional image data (depth map and segmentation map) is converted into a three-dimensional point cloud. Specifically, the camera intrinsic parameter matrix is used to back-project the pixels into three-dimensional space, with the following formula: P(x,y,z,class)=unproj[Depth(u,v,class)];
[0056] Here, (u,v) represents the pixel position in the image coordinate system, Depth(u,v) is the depth value of that pixel, class(u,v) is the semantic category of that pixel, and unproj() calculates the 3D coordinates (x,y,z) of the pixel in the camera coordinate system. Since each pixel carries a category label, each point P(x,y,z) in the point cloud also inherits semantic information, forming a 3D point cloud with semantic information, which represents the 3D structural information of the conference scene. Furthermore, since depth estimation may have errors, bilateral filtering or statistical filtering can be used to remove outliers. The TSDF fusion method can also be used for global optimization of the depth data, making the point cloud more stable.
[0057] Secondly, having obtained the 3D point cloud data of the meeting scene from the previous steps, we need to project the 3D point cloud onto a top-view plane (i.e., the plane formed by the X and Y axes) to obtain the 2D point cloud data P. floor (x,y) represents the planar projection of the scene, and its transformation formula is as follows: P floor (x,y) = Project(P(x,y,z));
[0058] Where P(x,y,z) is the 3D point cloud data, P floor (x,y) represents two-dimensional point cloud data, which contains all the information of the ground area of the conference scene.
[0059] Since point cloud data may contain noise (such as mismeasured points or missegmented regions), it is necessary to remove obvious outliers using statistical filtering or outlier detection algorithms (such as DBSCAN). For computational efficiency, voxel grid filtering can also be used to simplify the point cloud data and reduce the number of points in the cloud.
[0060] Subsequently, the RANSAC (Random Sample Consensus) algorithm is used to obtain the contour information of the meeting scene. Understandably, in RANSAC, planar models or linear models (such as line fitting to represent polygon edges) can be used to fit polygon or line boundaries. The line or polygon model obtained through RANSAC fitting generates the boundary contour of the meeting scene. For complex polygon contours, piecewise linear fitting can be used to divide the scene contour into multiple line segments to more accurately describe the edges of the meeting space. Additionally, geometric algorithms (such as solving for the minimum bounding rectangle of the polygon) can be used to further optimize and extract the geometric boundaries of the contour. Finally, the meeting scene contour information extracted by RANSAC will be represented in polygon form, which can be further used for subsequent geometric analysis and spatial layout optimization. This contour information can be used for: defining scene boundaries, dynamic analysis of meeting space, and interior planning or zoning, etc.
[0061] Finally, the geometric information corresponding to objects in the meeting scene is obtained, specifically the boundary information of the objects. This means extracting individual objects in the meeting room, such as the conference table, chairs, and projection screen. First, the point cloud is segmented by category using semantic information; for example, the point cloud of the conference table is P. table (x,y,z), the point cloud of the seat is P chair (x, y, z). Clustering algorithms (such as DBSCAN) can also be used to segment multiple instances of the same category (e.g., multiple seats). Next, the point cloud of the object is projected onto the top-view plane (XY axis), using the following formula: P object (x,y)=Project(P object (x,y,z));
[0062] The minimum bounding rectangle (MBR) is used to calculate the bounding box of an object, and the formula is as follows: MBR object =MinBoundingRect[P object [(x,y)];
[0063] MinBoundingRect[] is a method or function for calculating the minimum bounding rectangle.
[0064] Subsequently, the returned minimum bounding rectangle is the boundary information of the object, which consists of the four corner points of the rectangle or the width and height of the rectangle. It can contain the following information: the coordinates of the lower left corner of the rectangle, the coordinates of the upper right corner of the rectangle, the width and height of the rectangle, and the rotation angle.
[0065] This embodiment successfully reconstructs a static meeting scene by combining depth maps and segmentation maps, generating rich 3D structural information, contour information, and object geometric information. This process not only provides accurate 3D semantic point clouds for the meeting space but also effectively improves the scene's visualization and interactive performance by fusing semantic information and object boundary data. By accurately reconstructing the static layout of the meeting space and the geometric features of objects, users can obtain a high-fidelity meeting environment in a virtual space, helping to enhance the realism and immersion of the meeting experience. Furthermore, through efficient computation and low-frequency operation, excessive consumption of computing resources is avoided, enabling the system to maintain a low operating burden in practical applications and ensuring real-time performance and stability in complex scenarios.
[0066] In some embodiments, obtaining the contour information of a meeting scene based on a 3D point cloud includes: obtaining the point cloud of a target plane based on the 3D point cloud, and calculating the minimum convex polygon corresponding to the point cloud projected onto the target plane; connecting the vertices of the minimum convex polygon corresponding to the point cloud to generate the contour information of the meeting scene.
[0067] First, based on the above description of step S12, the two-dimensional point cloud data P can be obtained. floor (x,y). Secondly, the minimum convex polygon is calculated using the convex hull algorithm. The convex hull is the smallest convex shape containing all points, which can be achieved by calculating the boundary of these points. The calculation formula is as follows: Polygon(x,y)=ConvexHull[P floor [(x,y)];
[0068] Finally, once the vertices of the smallest convex polygon are obtained, these vertices are connected to form a closed polygon, representing the outline of the meeting scene. The boundaries formed by these vertices will accurately describe the spatial edges of the meeting scene, thus providing the necessary structural information for subsequent spatial understanding and virtual display.
[0069] In this embodiment, by using the convex hull algorithm to calculate the minimum convex polygon of the point cloud on the top-view plane and generate its contour, not only can the spatial boundaries of objects in a meeting scene be accurately described, but also reliable data support can be provided for further virtual meeting space modeling. This method can effectively handle complex 3D scenes, ensuring that the spatial layout of the meeting space is realistically and accurately digitally represented, enabling the digital twin system to more realistically reconstruct the meeting scene and enhance the user's immersive experience in the virtual space. The target plane is the top-view plane, i.e., the plane formed by the XY axes. The 3D point cloud (points in the XYZ coordinate system) is projected onto the XY-axis plane, ignoring changes in the Z-axis (height) direction and retaining only the horizontal position information. Then, convex hull calculation is performed on this 2D plane to obtain a closed minimum convex polygon, which is used to describe the contour boundary of the meeting scene from the top-view perspective. Optionally, the target plane can also be a screen with other perspectives, such as the plane formed by the XZ axes or the plane formed by the YZ axes. Therefore, the target plane can be understood as the reference plane on which the point cloud is to be projected and the contour calculated.
[0070] In some embodiments, the method for obtaining the contour information of a meeting scene based on a 3D point cloud may further include calculating the main orientation of the point cloud through PCA (Principal Component Analysis) dimensionality reduction and extracting the straight line boundary of the meeting room based on straight line fitting, thereby obtaining the contour information of the meeting scene.
[0071] The above steps achieve a static reconstruction of the meeting scene, obtaining its outline information. This outline can be presented from a top-down perspective, showcasing the overall shape of the meeting scene. After obtaining this top-down outline, users can freely adjust the shape and size of the polygons to flexibly define and edit the meeting area. Here, the polygon refers to the outline of the meeting scene as seen from a top-down perspective; this outline is simplified or represented as a polygon, representing the shape of the meeting area on a plane. Users can adjust the vertices and boundaries of this polygon to flexibly modify the layout of the meeting area.
[0072] S13. Based on the target image, obtain the dynamic person data information corresponding to the person in the meeting scene. The dynamic person data information includes the person's position information and depth information.
[0073] Depth information refers to the distance of an object or point relative to an observer (such as a camera or sensor) in three-dimensional space. Depth information helps determine the precise location of people in three-dimensional space, enhancing scene understanding; combined with depth information, it allows for the analysis of people's behavioral patterns, such as walking, standing still, or interacting, leading to behavior recognition and classification. Location information refers to the specific coordinates of a dynamic person in three-dimensional space.
[0074] The process involves obtaining dynamic character data information corresponding to individuals in a meeting scene based on the target image. This dynamic character data information includes the character's position and depth information, including: detecting head bounding boxes based on the target image (the head bounding boxes are used to identify the head regions corresponding to individuals in the meeting scene); obtaining the width and height of the head bounding boxes; inputting the width and height of the head bounding boxes into a depth calculation model to obtain the depth information of individuals in the meeting scene; performing back projection calculations based on the depth information and the center point coordinates of the head bounding boxes to obtain 3D point cloud data of individuals in the meeting scene; converting the 3D point cloud data to the camera coordinate system corresponding to the target image to generate a camera coordinate system point cloud; and extracting the x and y coordinates of the points corresponding to the head bounding boxes, the x and y coordinates of the points corresponding to the head bounding boxes, and the depth information of the individuals from the camera coordinate system point cloud to constitute the character's position information.
[0075] To obtain the feature information of head bounding boxes, a target image of the meeting scene is acquired via a camera. This image contains people and other objects within the meeting scene and is typically in RGB format. After acquiring the target image, an object detection algorithm (such as YOLO, Faster R-CNN, SSD, etc.) is used to detect head bounding boxes. The head bounding box identifies the head region of each person, and each bounding box includes: the coordinates of the top-left corner, the coordinates of the bottom-right corner, the width of the box, and its height. Subsequently, the width *w* and height *h* of each head bounding box are recorded and used as input parameters for depth estimation.
[0076] To calculate the depth information of a person, the width *w* and height *h* of the person's head bounding box are input to the depth calculation model. This depth calculation model can be a neural network-based depth estimation method, such as MiDaS, DPT (Depth Prediction Transformer), or a distillation learning model. This depth calculation model is already trained and can infer the person's depth in 3D space based on the size of the head bounding box. Understandably, the depth calculation model returns a depth value *D* for each person (representing the distance of the person from the camera). This depth value will be used in conjunction with point cloud data to further calculate the person's precise position.
[0077] Next, based on the depth information and the center point coordinates of the aforementioned head bounding boxes, back-projection calculations are performed to obtain the 3D point cloud data of the figures in the meeting scene. The center point coordinates of the obtained head bounding boxes can be represented by the midpoints of the top-left and bottom-right corners of the bounding box, yielding their position on the 2D image plane, which can be represented by (x, y). Combining this with the corresponding depth D, the point (X, Y, Z) in 3D space is reconstructed. This process can be based on the camera's intrinsic parameter matrix, then using the back-projection formula to calculate (X, Y, Z). The image coordinates (x, y) can be increased by 1 to form homogeneous coordinates (x, y, 1), normalized using the inverse of the camera's intrinsic parameter matrix, and then multiplied by the depth value D to obtain the actual 3D coordinates (X, Y, Z). Performing this back-projection once for each detected head bounding box center point yields the corresponding 3D position. If there are multiple figures, multiple 3D points will be obtained, forming the 3D point cloud data of each figure. The positional information of the figures in the meeting scene can be obtained through this 3D point cloud data.
[0078] This embodiment, by combining target images, depth calculation models, and point cloud data, achieves high-precision person detection and 3D localization. This embodiment boasts higher depth estimation accuracy, effectively reducing errors and ensuring precise fusion of spatial data. This step not only enhances meeting scene perception capabilities but also provides strong support for applications such as digital twin modeling, remote conferencing interaction, and behavior analysis, enabling intelligent meeting systems to possess a higher level of perception and interaction capabilities.
[0079] In some embodiments, the method further includes: obtaining a depth calculation model; obtaining the depth calculation model includes: detecting head bounding boxes based on the target image, the head bounding boxes being used to identify the head regions corresponding to people in the meeting scene; obtaining the width and height of the head bounding boxes; obtaining 3D point cloud data corresponding to the meeting scene using a LiDAR; converting the 3D point cloud data to the camera coordinate system corresponding to the target image to generate a camera coordinate system point cloud; mapping the camera coordinate system point cloud to the image coordinate system to obtain a sparse depth map of the meeting scene, the sparse depth map referring to a depth map in which only some pixels in the image have depth values; obtaining pixels with depth values based on the head bounding boxes and the sparse depth map, and selecting the minimum depth value as the depth of the head bounding box; calculating target parameters based on the depth of the head bounding boxes, as well as the width and height of the head bounding boxes; and determining the depth calculation model based on the calculated target parameters.
[0080] In some embodiments, obtaining pixels with depth values based on the head bounding box and the sparse depth map, and selecting the minimum depth value as the depth of the head bounding box, includes: traversing the region of the head bounding box and extracting the coordinates of the pixels within the head bounding box; finding the depth value corresponding to each pixel in the sparse depth map based on the coordinates of each pixel within the head bounding box, and filtering to obtain pixels with valid depth values; obtaining the minimum depth value of the pixels based on the pixels with valid depth values, and using the minimum depth value as the depth of the head bounding box.
[0081] As described above, the width *w* and height *h* of the head frame are used as input to a depth calculation model to obtain the person's depth information. This depth calculation model can be a neural network-based depth estimation method, such as MiDaS, DPT (Depth Prediction Transformer), or a distillation learning model. It should be noted that, in addition to using existing depth calculation models, the depth calculation model mentioned in this embodiment can also be used to obtain the person's depth information.
[0082] First, head bounding boxes are detected in the target image. This involves capturing the meeting scene with a camera to obtain the target image (RGB image). Object detection algorithms (such as YOLO, Faster R-CNN, SSD, etc.) are then used to detect the bounding boxes of each person in the meeting scene. The top-left corner coordinates of the head bounding box are (x...). min ,y min The coordinates of the bottom right corner of the head frame are (x max ,y max The width of the head frame is w = x. max -x min The height of the head frame is h = y max -y min .
[0083] Secondly, 3D point cloud data is acquired, and a point cloud in camera coordinate system is generated. This can be understood as follows: the LiDAR in the meeting scene scans the entire scene, collecting point cloud data in the LiDAR coordinate system. This point cloud data is stored as (x, y, z) coordinates, with each point representing an object or person in the meeting scene. Because the LiDAR and camera are installed in different locations, an extrinsic parameter matrix is required. The coordinate transformation formula is as follows:
[0084] Among them, P lidar and P camera These are point cloud data in radar coordinate system and camera coordinate system, respectively. This represents the transformation from the radar coordinate system to the camera coordinate system, where R is the rotation matrix and t is the translation vector.
[0085] Next, the point cloud is projected onto the image coordinate system to generate a sparse depth map. Understandably, this uses a camera projection model with the following formula: (u,v)=proj(P camera ); Depth[u,v]=P camera [Z];
[0086] Here, (u,v) represents the image index, and the corresponding index is assigned the Z-axis value of the camera coordinate system. The Z-axis is the forward orientation of the camera and represents the distance from the object to the camera. Because the radar has 128 lines, it only covers a portion of the pixels after being projected onto the image, resulting in a sparse depth map.
[0087] Next, the depth of the head bounding boxes is calculated. This can be understood as iterating through all head bounding boxes in the target image, and for each box: extracting the pixel coordinates (u, v) within the head bounding box. Then, finding the corresponding depth value Z in the sparse depth map. depth Filter out pixels with valid depth values. Pixels with valid depth values include background depth points and head depth points; head depth points are pixels with valid depth values, while background depth points are pixels with invalid depth values. Select the minimum depth value from the valid depth values as the depth of the head bounding box, using the formula: y = min(Z). depth This method can exclude background depth points, ensuring that the selected data is the depth information of the person.
[0088] Finally, the target parameters are calculated to determine the depth calculation model. Understandably, a mathematical model can be established based on the width w, height h, and depth y of the head bounding box, with the following formula: x w×h =w×h;
[0089] Wherein, the area of the head frame is x w×h The target parameters, a1 and a2, can be solved using multiple sets of data points. It's understandable that during the mathematical modeling phase, the target parameters (a1 and a2) can be pre-calculated using a batch of data to obtain the depth calculation model. In practical applications, the product of the width and height of the head bounding box obtained after target detection (x...) can be directly calculated. w×h Using this as input to the depth calculation model, the depth of the head frame can be calculated.
[0090] This embodiment achieves high-precision depth estimation of people by fusing image detection and LiDAR point cloud data. A sparse depth map is generated using camera coordinate transformation and image projection to ensure accurate matching of depth information with the target image. Simultaneously, target parameters are calculated using the width, height, and depth data of the head bounding box to obtain a depth calculation model, making it adaptable to different scenarios and enhancing its versatility and robustness. Furthermore, a minimum depth filtering strategy is employed to effectively remove background interference, improve ranging accuracy, and enhance the robustness of depth estimation to a certain extent, enabling the system to stably provide accurate depth information of people under different lighting, occlusion, or complex environments. Overall, this improves the accuracy and stability of person depth perception, providing a reliable foundation for spatial understanding and interactive applications in intelligent conferencing systems.
[0091] In some embodiments, after inputting the width and height of the head frame into the depth calculation model to obtain the depth information of the person in the meeting scene, the method further includes: based on a set sliding window, continuing to obtain a preset number of depth information, each depth information being obtained through the depth calculation model; calculating the median or average of the depth information obtained within the sliding window, and using the median or average as the new depth information of the person in the meeting scene.
[0092] First, the initial depth information is calculated. It is understood that the initial depth information can be obtained from the depth calculation model described above.
[0093] Secondly, a sliding window is used for temporal fusion. It's understandable that because the size of the head frame dynamically changes, single-frame depth estimation may fluctuate. Therefore, temporal fusion is introduced to improve the stability of depth estimation. Specifically, this involves: setting a sliding window with a size of n (usually an odd number to ensure symmetry); continuously acquiring depth information of the person in n consecutive frames and storing it within the sliding window; calculating the median or mean of the depth values within the sliding window: if the data is significantly affected by extreme values, the median is used as the final depth. Otherwise, the average value is calculated as the final depth, using the following formula: y i =(y i-n / 2 +…+y i-1 +y i +y i+1 +…+y i+n / 2 ) / n;
[0094] When n is odd, n / 2 is rounded down. It is understandable that temporal fusion can reduce errors caused by noise or sudden changes, thus improving the stability of depth estimation.
[0095] In addition, when a person is located at the edge of an image, the human head bounding box may only cover part of the human face (such as half of the face), resulting in the width of the human head bounding box being smaller than normal, which affects depth estimation. To address this, a fitting curve based on the height of the human head bounding box may be adopted in the edge range area, and the formula of the fitting curve is as follows:
[0096] where thre1 and thre2 are preset width boundary ranges of the image, x c is the center of the human head bounding box.
[0097] where the corresponding fitting curve is selected by determining whether the center x c of the human head bounding box is located in the edge area. If x c is within the boundary range (x c <thre1 or x c >thre2), the depth is calculated based on the height h of the human head bounding box. If x c is within the normal range, the product of the width and height of the human head bounding box x w×h is still used as the input to calculate the depth. It can be understood that by selecting different fitting curves according to the actual situation, relatively accurate depth estimation can still be ensured under different shooting angles, occlusion situations or edge scenes.
[0098] Finally, after time sequence fusion and edge scene optimization, the calculated new depth value is taken as the final output and used for subsequent analysis. The depth information can be used as the input of intelligent applications such as person tracking, interaction and behavior analysis, so as to improve the overall stability and accuracy of the system.
[0099] In this embodiment, the time sequence fusion makes the person depth estimation more stable, reduces the fluctuation caused by single-frame detection error, and thus improves the robustness of distance measurement. At the same time, when a person is located at the edge of the image, the distance measurement curve is dynamically adjusted to adapt to different scenes, so that the depth estimation is more accurate. On the whole, this method optimizes the person depth perception in the conference scene, ensures high-precision and high-reliability depth information in a dynamic environment, and provides strong support for behavior analysis, intelligent interaction and visual understanding.
[0100] S14. Fusing the static scene information and the dynamic person data information to generate a conference space perception result of the conference scene.
[0101] The step of fusing the static scene information and the dynamic person data information to generate the conference space perception result of the conference scene includes: fusing the dynamic person data information with the three-dimensional structure information of the conference scene, the contour information of the conference scene and the geometric information corresponding to objects in the conference scene to generate the conference space perception result of the conference scene, wherein the conference space perception result includes real-time position information and real-time activity information of the person in the conference scene, as well as environmental features of the conference scene.
[0102] First, the static information and dynamic person data of the meeting scene can be obtained from the above description. The static information includes, but is not limited to, 3D structural information, contour information, and object geometry information. The dynamic person data includes, but is not limited to, the person's position and depth information.
[0103] Secondly, after acquiring static scene information and dynamic character data, the following fusion operations are performed: Fusion of dynamic character data with static scene 3D structure information: Combining the position data of each character with the 3D structure in the scene to update the real-time position of the character in 3D space. Fusion of dynamic character data with contour information: Ensuring that the character's position in the scene does not conflict with the scene's contour, and correcting the character's spatial position information based on the scene's contour, especially in scenes with many people. Fusion of dynamic character data with object geometry information: Determining the relative positions of characters and objects to avoid conflicts between characters and objects, especially in scenes requiring spatial planning (such as adjusting seating or during interactions).
[0104] Finally, by fusing the above information, the meeting space perception result is obtained. The process of obtaining this meeting space perception result can also be considered as constructing a data twin model of the meeting space, which reflects the virtual data mapping of the people and environmental states in the meeting scene. This data twin model includes the following elements:
[0105] Real-time location information of personnel: describes the specific location of personnel in the meeting scene, and supports multi-person location tracking.
[0106] Real-time activity information of the character: describes the character's current state or actions, such as whether they are speaking, moving, or stationary.
[0107] Environmental features of the meeting scene: Combining 3D structure, contour and object geometry information, it provides a detailed spatial layout of the scene, supporting the comparison and simulation of virtual and reality.
[0108] It should be noted that as the meeting progresses, dynamic personnel information and environmental changes are constantly updated. Therefore, the data twin model needs to be updated in real time. Time-series data processing and dynamic tracking technologies can be used to ensure the accuracy and timeliness of the data twin model of the meeting space, reflecting real-time scene changes.
[0109] In this embodiment, a data twin model of the meeting space is generated by effectively integrating static scene information with dynamic participant data. This data twin model can reflect the location, activity status, and environmental characteristics of participants in the meeting scene in real time, providing accurate data support for meeting management, behavior analysis, space optimization, and interactive design. It not only improves the efficiency of space utilization during meetings but also supports application scenarios such as intelligent interaction and remote participant analysis. Through real-time spatial data feedback, the organizational efficiency, participation, and interactivity of meetings can be greatly enhanced.
[0110] It should be noted that in the above embodiments, there is no necessarily a certain order between the steps. Those skilled in the art can understand from the description of the embodiments of this application that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in turn, etc.
[0111] Please refer to Figure 3, which is a schematic diagram of the structure of a meeting space perception device provided in an embodiment of this application. The device 20 includes: an image acquisition module 21, a static scene reconstruction module 22, a dynamic person data information acquisition module 23, and a perception fusion module 24.
[0112] The image acquisition module 21 is used to acquire the target image corresponding to the meeting scene.
[0113] The static scene reconstruction module 22 is used to reconstruct the meeting scene based on the target image and generate static scene information of the meeting scene. The static scene information includes the three-dimensional structural information of the meeting scene, the contour information of the meeting scene, and the geometric information of the objects in the meeting scene.
[0114] The dynamic person data acquisition module 23 is used to acquire dynamic person data information corresponding to the person in the meeting scene based on the target image. The dynamic person data information includes the person's position information and depth information.
[0115] The perception fusion module 24 is used to fuse static scene information and dynamic person data information to generate a meeting space perception result for the meeting scene.
[0116] The aforementioned meeting space sensing device 20 can be a software module, which includes several instructions stored in a memory. The processor can access the memory, call the instructions, and execute them to complete the meeting space sensing method described in the above embodiments.
[0117] In some embodiments, the aforementioned conference space sensing device 20 can also be constructed from hardware devices. For example, the conference space sensing device 20 can be constructed from one or more chips, and the chips can work in coordination to complete the conference space sensing method described in the various embodiments. As another example, the aforementioned conference space sensing device 20 can also be constructed from various logic devices, such as general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontrollers, ARM (Acorn RISC Machine) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination of these components.
[0118] It should be noted that the aforementioned meeting space sensing device 20 can execute the meeting space sensing method for electronic devices provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in the embodiments of the meeting space sensing device 20 can be found in the aforementioned meeting space sensing method for electronic devices provided in the embodiments of this application.
[0119] Please refer to Figure 4, which is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 30 includes one or more processors 31 and a memory 32. The memory 32 is connected to one or more processors 31, for example, via a bus.
[0120] Processor 31 is configured to support the electronic device 30 in performing the corresponding functions described in the method embodiments. Processor 31 may be a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof. The aforementioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.
[0121] Memory 32 is used to store program code, etc. Memory 32 may include volatile memory (VM), such as random access memory (RAM); memory may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory 32 may also include combinations of the above types of memory.
[0122] The memory 32 can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the conference space perception method in the embodiments of this application. The processor 31 executes various functional applications and data processing of the conference space perception method and conference space perception device by running the non-volatile software programs, instructions, and modules stored in the memory 32, that is, it realizes the functions of the various modules or units of the conference space perception method and conference space perception device provided in the above method embodiments.
[0123] The memory 32 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function. The data storage area may store data created based on the use of the resource dynamic scheduling device. In some embodiments, the memory 32 may include memory remotely located relative to the processor 31, and such remote memory may be connected to the resource dynamic scheduling device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0124] The one or more modules are stored in the memory 32. When executed by the one or more processors 31, they execute the meeting space awareness method in any of the above method embodiments. For example, they execute the method steps described in the above method embodiments to realize the functions of the modules described in the above device embodiments.
[0125] The electronic device in this application embodiment can be a conference tablet with camera functionality, or a system composed of other firmware with camera functionality (such as a videobar) and a corresponding processing module (such as an RKNN3588 module, a laptop computer, etc.).
[0126] This application provides a non-volatile computer-readable storage medium storing computer-executable instructions that can be executed by one or more processors, such as processor 31 in FIG4, enabling the one or more processors 31 to execute the meeting space awareness method in any of the above method embodiments.
[0127] This application provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions that, when executed by the electronic device, enable the electronic device to perform the meeting space awareness method in any of the above method embodiments.
[0128] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.
[0129] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.
Claims
1. A method for perceiving meeting space, characterized in that, include: Obtain the target image corresponding to the meeting scene; The meeting scene is reconstructed statically based on the target image to generate static scene information of the meeting scene. The static scene information includes the three-dimensional structure information of the meeting scene, the outline information of the meeting scene, and the geometric information of the objects in the meeting scene. Based on the target image, obtain dynamic character data information corresponding to the characters in the meeting scene, wherein the dynamic character data information includes the position information and depth information of the characters; The static scene information and the dynamic character data information are fused to generate the meeting space perception result of the meeting scene.
2. The method according to claim 1, characterized in that, The step of reconstructing the meeting scene based on the target image to generate static scene information of the meeting scene includes: Based on the target image, obtain the depth map and segmentation map of the meeting scene; Based on the depth map and the segmentation map, a 3D point cloud with semantic information is generated, and the 3D point cloud with semantic information is the 3D structural information of the conference scene; Based on the three-dimensional point cloud, the contour information of the meeting scene is obtained; Based on the semantic information of the 3D point cloud, the point cloud of the object in the meeting scene is extracted, the point cloud of the object is projected onto the target plane, and the boundary information of the object is calculated based on the target plane projected by the point cloud of the object.
3. The method according to claim 2, characterized in that, Obtaining the contour information of the meeting scene based on the 3D point cloud includes: Based on the three-dimensional point cloud, the point cloud of the target plane is obtained, and the minimum convex polygon corresponding to the point cloud projected onto the target plane is calculated. Connect the vertices of the smallest convex polygon corresponding to the point cloud to generate the contour information of the meeting scene.
4. The method according to claim 1, characterized in that, The step involves obtaining dynamic character data information corresponding to the characters in the meeting scene based on the target image. This dynamic character data information includes the character's position information and depth information, including: A head bounding box is detected based on the target image, and the head bounding box is used to identify the head region corresponding to the person in the meeting scene; Obtain the width and height of the head frame; The width and height of the head frame are input into the depth calculation model to obtain the depth information of the person in the meeting scene; Based on the depth information and the center point coordinates of the head frame, back projection calculation is performed to obtain the three-dimensional point cloud data of the person in the meeting scene; the three-dimensional point cloud data constitutes the position information of the person.
5. The method according to claim 4, characterized in that, The method further includes: obtaining the depth calculation model; The process of obtaining the depth calculation model includes: A head bounding box is detected based on the target image, and the head bounding box is used to identify the head region corresponding to the person in the meeting scene; Obtain the width and height of the head frame; The three-dimensional point cloud data corresponding to the meeting scene is obtained by LiDAR; The 3D point cloud data is converted to the camera coordinate system corresponding to the target image to generate a camera coordinate system point cloud. The point cloud in the camera coordinate system is mapped to the image coordinate system to obtain a sparse depth map of the conference scene; Based on the head frame and the sparse depth map, obtain pixels with depth values, and select the minimum depth value as the depth of the head frame; Calculate the target parameters based on the depth, width, and height of the head frame; The depth calculation model is determined based on the calculated target parameters.
6. The method according to claim 5, characterized in that, The step of obtaining pixels with depth values based on the head frame and the sparse depth map, and selecting the minimum depth value as the depth of the head frame, includes: Traverse the region of the head frame and extract the coordinates of the pixels within the head frame; Based on the coordinates of each pixel within the head frame, the depth value corresponding to the pixel is found in the sparse depth map, and pixels with valid depth values are selected. Based on the effective depth values of the pixels, the minimum depth value of the pixels is obtained, and the minimum depth value is used as the depth of the head frame.
7. The method according to claim 4, characterized in that, After the step of inputting the width and height of the head frame into the depth calculation model to obtain the depth information of the person in the meeting scene, the method further includes: Based on the set sliding window, a preset number of depth information is obtained, and each preset number of depth information is obtained through the depth calculation model; Calculate the median or average of the depth information obtained within the sliding window; The depth information of the people in the meeting scene is updated based on the median or the average value.
8. The method according to claim 1, characterized in that, The step of fusing the static scene information and the dynamic person data information to generate the meeting space perception result of the meeting scene includes: The dynamic character data information is fused with the three-dimensional structural information of the meeting scene, the contour information of the meeting scene, and the geometric information corresponding to the objects in the meeting scene to generate the meeting space perception result of the meeting scene. The meeting space perception result includes the real-time position information and real-time activity information of the characters in the meeting scene, as well as the environmental features of the meeting scene.
9. A conference space sensing device, characterized in that, include: The image acquisition module is used to acquire target images corresponding to the meeting scene; The static scene reconstruction module is used to reconstruct the meeting scene based on the target image, and generate static scene information of the meeting scene. The static scene information includes the three-dimensional structural information of the meeting scene, the contour information of the meeting scene, and the geometric information corresponding to the objects in the meeting scene. The dynamic person data information acquisition module is used to acquire dynamic person data information corresponding to the person in the meeting scene based on the target image. The dynamic person data information includes the person's position information and depth information. The perception fusion module is used to fuse the static scene information and the dynamic person data information to generate the meeting space perception result of the meeting scene.
10. An electronic device, characterized in that, include: A memory and a processor, the memory being connected to the processor, the processor being configured to execute one or more computer programs stored in the memory, the processor, when executing the one or more computer programs, causing the electronic device to perform the method as described in any one of claims 1-8.
11. A non-volatile computer-readable storage medium, characterized in that, The non-volatile computer-readable storage medium stores computer-executable instructions that, when executed by an electronic device, cause the electronic device to perform the method described in any one of claims 1-8.
12. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions that, when executed by an electronic device, cause the electronic device to perform the method of any one of claims 1-8.
