Image sensor data control system

Abstract

The present invention addresses the problem of providing an image sensor data processing system capable of processing, at high speed, image sensor data that is composed of a point cloud in a real space. The present invention comprises: a plurality of sensor devices 1 disposed on the environment side of a real space; and a management device 2 connected to the sensor devices 1 in a communicable state. Each of the sensor devices 1 comprises: a sensor unit 11 that sequentially acquires an image sensor data item that is composed of a point cloud in a prescribed real space; and a transmission unit 12 that transmits a frame of the image sensor data item that is composed of the point cloud in the prescribed real space to the management device 2 at a prescribed transmission cycle. The management device 2 comprises a composite unit 22 that composites the frames of the image sensor data items that are each composed of a point cloud in the prescribed real space and are transmitted from the sensor devices 1, while arranging the frames in order of transmission timing of the sensor devices 1.

Description

Image Sensor Data Control System

[0001] The present invention relates to an image sensor data control system that processes image sensor data acquired in the real space at high speed.

[0002] In recent years, as a global trend towards rapid urbanization, a significant growth in the market for autonomous driving of dynamic objects such as automobiles, micromobility, drones, and transport robots is predicted. Research is being conducted to track dynamic objects by utilizing a sensor network equipped with sensors placed on the environmental side of the real space (see, for example, Non-Patent Documents 1 and 2).

[0003] As a technology for sensing dynamic objects in a predetermined real space, a sensor called LiDAR (light detection and ranging) has attracted attention. This LiDAR is a type of sensor that uses laser light. Compared to radio waves, it has a higher radiation beam density. By scanning and irradiating an object with a laser light of a short wavelength, in addition to the distance to the object, the position and shape of the object can be accurately detected. Therefore, when LiDAR is used in a real space where static objects such as walls and dynamic objects such as automobiles exist, three-dimensional information of the real space can be acquired as one frame of image sensor data consisting of a point cloud of laser light for each scan.

[0004] At this time, in order to accurately sense a predetermined real space, a plurality of LiDARs (L1 to L4) are respectively arranged at different locations on the environmental side of the real space, and image sensor data consisting of a point cloud is acquired from various angles with respect to the predetermined real space. The frame F of the image sensor data consisting of the point cloud in the predetermined real space acquired by each LiDAR (L1 to L4) is randomly transmitted from each LiDAR (L1 to L4) to an edge computer (management device) at a predetermined transmission cycle T as shown in FIG. 6. After a new frame F' is generated by performing calibration in the edge computer, it is used in a subsequent digital twin (virtual space) system or AI (artificial intelligence) system.

[0005] K. Akiyama et al., “Edge computing system with multi-lidar sensor network for robustness of autonomous personal-mobility,” pp. 290-295, 2022. T. Kudo et al., “Edge system with multi-lidar sensor network for tracking micro-mobility vehicles,” pp. 983-984, 2023.

[0006] However, the scanning performance of a general-purpose LiDAR is generally around 10 Hz. Frames F of image sensor data consisting of point clouds in a predetermined real space, acquired by each LiDAR (L1 to L4), are transmitted from each LiDAR (L1 to L4) to the edge computer at a transmission period T (1 / 10 second). Therefore, as shown in Figure 6(b), the edge computer aggregates the frames F of image sensor data consisting of point clouds transmitted from each LiDAR (L1 to L4) and generates a new frame F' approximately every 1 / 10 second, the same as the transmission period T. This poses a problem in later digital twin systems and AI systems, where high-speed processing of dynamic objects is particularly required, as the data transmission process cannot keep up. Of course, it is conceivable to use LiDARs with higher scanning performance (e.g., 30 Hz), but such high-performance LiDARs are very expensive, leading to the problem of high costs when multiple units are installed.

[0007] The present invention has been made in view of the above-mentioned problems, and aims to provide an image sensor data processing system that can process image sensor data consisting of point clouds in real space at high speed.

[0008] To achieve the above objective, the present invention comprises a plurality of sensor devices arranged on the environment side of a real space, and a management device connected to each sensor device in a communicative manner, wherein each sensor device includes a sensor unit that sequentially acquires image sensor data consisting of point clouds in a predetermined real space by scanning a predetermined real space including dynamic objects, and a transmission unit that transmits frames of image sensor data consisting of point clouds in a predetermined real space acquired sequentially by the sensor unit to the management device at a predetermined transmission period, wherein the management device includes a receiving unit that receives each frame of image sensor data consisting of point clouds in a predetermined real space transmitted from each sensor device, a combining unit that combines each frame of image sensor data consisting of point clouds in a predetermined real space received by the receiving unit in the order of the transmission timing of each sensor device, a processing unit that performs predetermined processing on the frames of image sensor data consisting of point clouds in a predetermined real space that have been combined by the combining unit in the order of the transmission timing of each sensor device, and a scheduler unit that controls the transmission timing of frames of image sensor data consisting of point clouds in a predetermined real space for each sensor device.

[0009] Furthermore, the scheduler unit may control the transmission timing of frames of image sensor data consisting of point clouds in a predetermined real space for each sensor device by shifting it by T / N, where T is the frame transmission period for each sensor device and N is the number of sensor devices.

[0010] Furthermore, the processing unit may transmit each frame of the image sensor data, which consists of a point cloud in a predetermined real space and is synthesized by the synthesis unit in the order of the transmission timing of each sensor device, to a subsequent digital twin system or AI system.

[0011] Furthermore, the processing unit may perform processes for detecting and / or tracking dynamic objects.

[0012] Furthermore, the management device according to the present invention is a management device in an image sensor data control system comprising a plurality of sensor devices arranged on the environment side of real space and the management device connected to each sensor device in a communication-enabled state, characterized in that it comprises a receiving unit that receives each frame of image sensor data consisting of a point cloud in a predetermined real space transmitted from each sensor device, a combining unit that combines each frame of image sensor data consisting of a point cloud in a predetermined real space received by the receiving unit in the order of the transmission timing of each sensor device, and a processing unit that performs predetermined processing on the frames of image sensor data consisting of a point cloud in a predetermined real space that have been combined by the combining unit in the order of the transmission timing of each sensor device.

[0013] Furthermore, the computer program according to the present invention is a computer program used in the image sensor data control system comprising a plurality of sensor devices arranged on the environment side of real space and a management device connected to each sensor device in a communication-enabled state, wherein the sensor devices are configured to function as a sensor unit that sequentially acquires image sensor data consisting of point clouds in a predetermined real space by scanning a predetermined real space including dynamic objects, and a transmission unit that transmits frames of image sensor data consisting of point clouds in a predetermined real space acquired sequentially by the sensor unit to the management device at a predetermined transmission period, and the management device is configured to function as a receiving unit that receives each frame of image sensor data consisting of point clouds in a predetermined real space transmitted from each sensor device, a combining unit that combines each frame of image sensor data consisting of point clouds in a predetermined real space received by the receiving unit in the order of the transmission timing of each sensor device, a processing unit that performs predetermined processing on the frames of image sensor data consisting of point clouds in a predetermined real space that have been combined by the combining unit in the order of the transmission timing of each sensor device, and a scheduler unit that controls the transmission timing of frames of image sensor data consisting of point clouds in a predetermined real space for each sensor device.

[0014] Furthermore, the computer program according to the present invention is a computer program used in the management device in the image sensor data control system, which comprises a plurality of sensor devices arranged on the real-space environment side and the management device connected to each sensor device in a communicative state, characterized in that the management device functions as a receiving unit that receives each frame of image sensor data consisting of a predetermined point cloud in real space transmitted from each sensor device, a combining unit that combines each frame of image sensor data consisting of a predetermined point cloud in real space received by the receiving unit in the order of the transmission timing of each sensor device, a processing unit that performs predetermined processing on the frame of image sensor data consisting of a predetermined point cloud in real space that has been combined by the combining unit in the order of the transmission timing of each sensor device, and a scheduler unit that controls the transmission timing of the frame of image sensor data consisting of a predetermined point cloud in real space for each sensor device.

[0015] According to the present invention, each frame of image sensor data consisting of a point cloud in a predetermined real space, transmitted from each sensor device, is combined in order of the transmission timing of each sensor device. This enables high-speed processing of image sensor data consisting of a point cloud in real space, and consequently, enables efficient processing in digital twin systems and AI systems.

[0016] This is a schematic diagram showing the configuration of an image sensor data control system according to an embodiment of the present invention. This is a conceptual diagram showing the installation state of the sensor device in a predetermined real space. This is a diagram showing (a) the transmission state of frames in each sensor device and (b) the synthesis state of each frame in the management device in the image sensor data control system of Figure 1. This is a flowchart showing the operation of the image sensor data control system of Figure 1. This is a diagram showing (a) an example of image sensor data synthesis by a conventional system and (b) an example of image sensor data synthesis by the image sensor data control system of Figure 1. This is a diagram showing (a) the transmission state of frames in each sensor device and (b) the synthesis state of each frame in the management device in a conventional image sensor data control system.

[0017] Next, embodiments of the image sensor data control system according to the present invention (hereinafter referred to as "this system") will be described with reference to Figures 1 to 6.

[0018] [Overall Configuration] As shown in Figure 1, this system comprises a plurality of sensor devices 1 (four in this embodiment) installed on the environment side of a predetermined real space, and a management device 2 as an edge computer connected to each sensor device 1 in a state where communication is possible via a predetermined wired or wireless communication path. Image sensor data in the predetermined real space acquired by each sensor device 1 is transmitted from each sensor device 1 to the management device 2, synthesized in the management device 2, and then used in a subsequent digital twin (virtual space) system or AI (artificial intelligence) system.

[0019] Real space refers to existing, primarily three-dimensional spaces related to social life or the environment, such as roads, streets, buildings, indoors, rivers, and mountains. Real space typically includes dynamic objects that move or act within a given real space, such as automobiles, micromobility vehicles, drones, transport robots, and pedestrians, as well as static objects that remain stationary within a given real space, such as walls, roads, and furniture. However, it is assumed that at least dynamic objects are included.

[0020] In this embodiment, the digital twin (virtual space) primarily refers to a space in which the real physical space is synchronized in a timely manner with a three-dimensional digital physical space. Furthermore, the AI ​​system (artificial intelligence) is primarily used in systems that use machine learning to detect and track dynamic objects such as people and vehicles.

[0021] [Configuration of Sensor Device 1] As shown in Figure 2, the sensor device 1 is installed on a traffic light, security camera device, traffic camera device, dedicated support pole, etc., located in a predetermined real-world environment, and four sensor devices 1 (L1, L2, L3, L4) acquire point cloud image sensor data from different angles on the same real-world space. Specifically, as shown in Figure 1, the sensor device 1 includes a sensor unit 11 that acquires point cloud image sensor data in real space, and a transmission unit 12 that transmits frames F of the image sensor data in real space acquired by the sensor unit 11 to a management device 2.

[0022] The sensor unit 11 sequentially acquires image sensor data consisting of a point cloud in a predetermined real space, including dynamic objects, by scanning. In this embodiment, LiDAR (light detection and ranging) is used. LiDAR is a type of sensor that uses laser light, and compared to radio waves, it has a higher density of radiant flux. By irradiating an object with short-wavelength laser light while scanning it, it can accurately detect not only the distance to the object but also its position and shape. Therefore, when LiDAR is used in a real space where static objects such as walls and dynamic objects such as automobiles exist, three-dimensional information of the real space can be acquired with each scan by one frame F of image sensor data consisting of a point cloud of laser light.

[0023] The transmitting unit 12 transmits frames F of image sensor data consisting of point clouds in real space, acquired sequentially by the sensor unit 11, to the management device 2 at a predetermined transmission period T. For example, the scanning performance of a general-purpose sensor unit 11 (LiDAR) is approximately 10 Hz, and as shown in Figure 3(a), frames F of image sensor data consisting of point clouds in real space, acquired by the sensor unit 11, are transmitted from each sensor device 1 (L1 to L4) to the management device 2 at a transmission period T (1 / 10 second).

[0024] [Configuration of Management Device 2] The management device 2 is connected to each sensor device 1 via a wired or wireless communication path and is used as an edge computer that is close to each sensor device 1 in terms of communication distance. Specifically, as shown in Figure 1, the management device 2 includes a receiving unit 21 that receives each frame F of image sensor data consisting of point clouds in real space transmitted from each sensor device 1, a combining unit 22 that combines each frame F of image sensor data consisting of predetermined point clouds in real space received by the receiving unit 21, a processing unit 23 that performs predetermined processing on the frame F of image sensor data consisting of predetermined point clouds in real space that has been combined by the combining unit 22, and a scheduler unit 24 that controls the transmission timing of the frame F of image sensor data consisting of point clouds in real space to each sensor device 1.

[0025] When the synthesis unit 22 synthesizes each frame F (L1, L2, L3, L4, L1…) of the image sensor data consisting of point clouds in real space transmitted from each sensor device 1, it synthesizes each frame F (L1, L2, L3, L4, L1…) of the image sensor data consisting of point clouds in real space in the order of the transmission timing of each sensor device 1, as shown in Figure 3(b).

[0026] In this regard, conventionally, when synthesizing each frame F (L1, L2, L3, L4) of image sensor data consisting of point clouds in real space, as shown in Figure 6(b), calibration is performed on each frame F (L1, L2, L3, L4) of the image sensor data consisting of point clouds in real space with the aim of improving the accuracy of the image sensor data consisting of point clouds in real space. As a result, new frames F are synthesized every 1 / 10th of a second by overlapping them regardless of the transmission timing order of each sensor device 1, which meant that the data processing could not keep up, especially when high-speed processing of dynamic objects was required in later digital twin systems and AI systems.

[0027] In contrast, in the present invention, as described above, each frame F (L1, L2, L3, L4, L1…) of the image sensor data consisting of point clouds in real space is combined in order of the transmission timing of each sensor device 1. Therefore, each frame F (L1, L2, L3, L4, L1…) of the image sensor data consisting of point clouds in real space can be transmitted at high speed to a subsequent digital twin system or AI system at the same timing as the transmission timing of each frame F in each sensor device 1, enabling high-speed processing of dynamic objects.

[0028] The processing unit 23 performs predetermined processing on the image sensor data frames F (L1, L2, L3, L4, L1…) consisting of point clouds in real space, which have been synthesized by the synthesis unit 22 in the order of the transmission timings of each sensor device 1. In this embodiment, the processing unit 23 performs the processing to transmit the image sensor data frames F (L1, L2, L3, L4, L1…) consisting of point clouds in real space to a subsequent digital twin system or AI system.

[0029] The scheduler unit 24 controls the transmission timing of frames F of image sensor data consisting of point clouds in real space for each sensor device 1. In this embodiment, if the transmission period of frames F in each sensor device 1 is T and the number of sensor devices 1 is N, the scheduler unit 24 controls the transmission timing of frames F of image sensor data consisting of point clouds in real space for each sensor device 1 (L1 to L4) to be shifted by T / N. For example, if the transmission period T of frames F in each sensor device 1 is 1 / 10 second and the number of sensor devices 1 N is 4, the scheduler unit 24 controls the transmission timing of frames F of image sensor data consisting of point clouds in real space for each sensor device 1 (L1 to L4) to be shifted by T / 4 (1 / 40 second). For example, if the transmission timing of frame F(L1) of sensor device 1(L1) is t, then the transmission timing of frame F(L2) of sensor device 1(L2) will be t+T / 4, the transmission timing of frame F(L3) of sensor device 1(L3) will be t+2T / 4, and the transmission timing of frame F(L4) of sensor device 1(L4) will be t+3T / 4.

[0030] According to this, when the above-mentioned synthesis unit 22 synthesizes each frame F (L1, L2, L3, L4, L1…) of the image sensor data consisting of point clouds in real space, as shown in Figure 3(b), each frame F (L1, L2, L3, L4, L1…) of the image sensor data consisting of point clouds in real space is synthesized in order of the transmission timing T / 4 (1 / 40 second) of each sensor device 1. This allows each frame F (L1, L2, L3, L4, L1…) of the image sensor data consisting of point clouds in real space to be transmitted at high speed to the subsequent digital twin system or AI system at the same transmission timing T / 4 (1 / 40 second) as the transmission timing of each sensor device 1, enabling extremely high-speed processing.

[0031] [Transmission and Synthesis Flow of Image Sensor Data by This System] Next, the transmission and synthesis flow of image sensor data by this system will be explained using the flowchart shown in Figure 4. For the sake of explanation, frames F (L1, L2, L3, L4) of image sensor data consisting of point clouds in a predetermined real space acquired by the first scan of each sensor device 1 (L1 to L4) will be described.

[0032] First, in each sensor device 1 (L1 to L4), the sensor unit 11 sequentially acquires image sensor data consisting of point clouds in real space by scanning a real space containing a predetermined dynamic object. Then, the transmission unit 12 transmits frames F (L1, L2, L3, L4) of the image sensor data consisting of point clouds in real space acquired by the sensor unit 11 to the management device 2 at a predetermined transmission period T (S11 to S14). At this time, for example, as shown in Figure 3(a), each sensor device 1 (L1 to L4) is controlled by the scheduler unit 24 of the management device 2 to shift the transmission timing of frames F of the image sensor data consisting of point clouds in real space by T / 4 (1 / 40 second).

[0033] Next, in the management device 2, the receiving unit 21 receives each frame F (L1, L2, L3, L4) of the image sensor data consisting of point clouds in real space transmitted from each sensor device 1 at every T / 4 (1 / 40 second) of the transmission timing (S21-S24).

[0034] Then, as shown in Figure 3(b), the synthesis unit 22 synthesizes each frame F (L1, L2, L3, L4) of the image sensor data consisting of point clouds in real space transmitted from each sensor device 1 (L1 to L4) by arranging them in the order of the transmission timing of each sensor device 1 (S25).

[0035] Then, the processing unit 23 performs a process to transmit the image sensor data frames F (L1, L2, L3, L4), which consist of point clouds in real space synthesized by the synthesis unit 22 in the order of the transmission timings of each sensor device 1 (L1 to L4), to the subsequent digital twin system or AI system at T / 4 (1 / 40 second) intervals (S26).

[0036] <Example 1> This section describes an example of applying this system to monitoring for the purpose of preventing accidents involving vehicles traveling on a designated road.

[0037] In conventional systems, if the scanning performance of the sensor unit 11 (LiDAR) of each sensor device 1 is 10 Hz, the transmission period of frame F of image sensor data consisting of point clouds in a predetermined real space (road) transmitted from sensor device 1 to management device 2 is 1 / 10 second. Therefore, it takes 1 / 10 second for the management device 2 to generate a new frame F by calibration from the frame F of image sensor data consisting of point clouds in a predetermined real space (road) transmitted from each sensor device 1. As a result, if a vehicle is traveling at high speed of 50 km / h or more on the road, an error of 1.39 meters (50 km / h x 1 / 10 second) occurs. In particular, for vehicles with small body sizes such as small cars and motorcycles, an error of 1.39 meters can lead to significant detection errors and shape estimation errors.

[0038] On the other hand, in this system, if there are four sensor devices 1, and the scanning performance of the sensor unit 11 (LiDAR) of each sensor device 1 is 10 Hz, then by controlling the transmission timing of frames F of image sensor data consisting of point clouds in real space to be shifted by T / 4 (1 / 40 second) for each sensor device 1, the synthesis unit 22 can synthesize each frame F (L1, L2, L3, L4, L1…) of image sensor data consisting of point clouds in real space while arranging them in the order of the transmission timing T / 4 (1 / 40 second) of each sensor device 1. As a result, each frame F (L1, L2, L3, L4, L1…) of image sensor data consisting of point clouds in real space can be processed at high speed every T / 4 (1 / 40 second), reducing the error to 0.35 m (50 km / h × 1 / 40 second), and improving detection errors and shape estimation errors.

[0039] <Example 2> This section describes an example in which the system is applied to monitoring for the purpose of preventing accidents involving drones flying in a predetermined airspace.

[0040] In conventional systems, if the scanning performance of the sensor unit 11 (LiDAR) of each sensor device 1 is 10 Hz, the transmission period of frame F of image sensor data consisting of point clouds in a predetermined real space (air) transmitted from sensor device 1 to management device 2 is 1 / 10 second. Therefore, it takes 1 / 10 second for the management device 2 to generate a new frame F by calibration from the frame F of image sensor data consisting of point clouds in a predetermined real space (air) transmitted from each sensor device 1. As a result, if the drone is traveling at high speed in the air at 50 km / h or more, an error of 1.39 meters (50 km / h × 1 / 10 second) occurs. In particular, because drones have a small body size, an error of 1.39 m can lead to significant detection errors and shape estimation errors.

[0041] On the other hand, in this system, if there are four sensor devices 1, and the scanning performance of the sensor unit 11 (LiDAR) of each sensor device 1 is 10 Hz, then by controlling the transmission timing of frames F of image sensor data consisting of point clouds in real space to be shifted by T / 4 (1 / 40 second) for each sensor device 1, the synthesis unit 22 can synthesize each frame F (L1, L2, L3, L4, L1…) of image sensor data consisting of point clouds in real space while arranging them in the order of the transmission timing T / 4 (1 / 40 second) of each sensor device 1. As a result, each frame F (L1, L2, L3, L4, L1…) of image sensor data consisting of point clouds in real space can be processed at high speed every T / 4 (1 / 40 second), reducing the error to 0.35 m (50 km / h × 1 / 40 second), and improving detection errors and shape estimation errors.

[0042] <Example 3> This section describes an example in which the system is applied to the three-dimensional analysis of human dance movements in a predetermined room.

[0043] In conventional systems, if the scanning performance of the sensor unit 11 (LiDAR) of each sensor device 1 is 10 Hz, the transmission period of frame F of image sensor data consisting of point clouds in a predetermined real space (indoors) transmitted from sensor device 1 to management device 2 is 1 / 10 second. Therefore, it takes 1 / 10 second for management device 2 to generate a new frame F by calibration from the frame F of image sensor data consisting of point clouds in a predetermined real space (indoors) transmitted from each sensor device 1. As a result, if a person is moving their hand at a speed of 100 km / h or more, an error of 2.78 meters (100 km / h × 1 / 10 second) occurs. This error is very large compared to the thickness of a person's arm, and it is not possible to perform three-dimensional analysis of the movement of image sensor data containing this large error (see, for example, the example in Figure 5(a)).

[0044] On the other hand, in this system, when the number of sensor devices 1 is four and the scanning performance of the sensor unit 11 (LiDAR) of each sensor device 1 is 10 Hz, if the transmission timing of the frame F of the image sensor data composed of point clouds in the real space is controlled to be shifted by T / 4 (1 / 40 seconds) for each sensor device 1, the combining unit 22 can combine each frame F (L1, L2, L3, L4, L1...) of the image sensor data composed of point clouds in the real space while arranging them in the order of the transmission timing T / 4 (1 / 40 seconds) of each sensor device 1. Therefore, each frame F (L1, L2, L3, L4, L1...) of the image sensor data composed of point clouds in the real space can be processed at high speed every T / 4 (1 / 40 seconds), and the error is reduced to 0.69 m (100 km / h × 1 / 40 seconds), improving the accuracy of the three-dimensional analysis of human dance movements (see, for example, the actual example in FIG. 5(b)).

[0045] In this embodiment, when the transmission period of the frame F in each sensor device 1 is T and the number of sensor devices 1 is N, the scheduler unit 24 controls the transmission timing of the frame F of the image sensor data composed of point clouds in the real space to be shifted by T / N for each sensor device 1. However, the transmission timing may be controlled to be shifted at other timings. However, in order to execute high-speed processing of each frame F, it is preferable that the transmission timing of each frame F of the image sensor data is within the range of the transmission period T of the first frame F (L1 in this embodiment).

[0046] Also, although the scheduler unit 24 controls the transmission period T of the frame F in each sensor device 1, the transmission period T may be set in advance to be shifted in each sensor device 1.

[0047] Further, although the processing unit 23 transmits the image sensor data composed of point clouds in a predetermined real space to a later digital twin system or AI system, it may be transmitted to other systems, or the processing may be executed within the management device 2. Alternatively, the processing unit 23 may execute processing for detecting and / or tracking dynamic objects such as people and vehicles.

[0048] Also, when this system is configured by a computer program, the computer program according to the present invention is a computer program used in an image sensor data control system including a plurality of sensor devices 1 arranged on the environment side of the real space and a management device 2 connected to each sensor device 1 in a communicable state. This computer program causes the sensor device 1 to function as a sensor unit 11 that sequentially acquires image sensor data composed of a point cloud in a predetermined real space by scanning a predetermined real space including moving objects, and a transmission unit 12 that transmits a frame of the image sensor data composed of the point cloud in the predetermined real space sequentially acquired by the sensor unit 11 to the management device 2 at a predetermined transmission period. Further, this computer program causes the management device 2 to function as a reception unit 21 that receives each frame of the image sensor data composed of the point cloud in the predetermined real space transmitted from each sensor device 1, a synthesis unit 22 that synthesizes each frame F of the image sensor data composed of the point cloud in the predetermined real space received by the reception unit 21 in the order of the transmission timings of each sensor device 1, and a processing unit 23 that executes a predetermined process (for example, transmission to another system, etc.) on the frame F of the image sensor data composed of the point cloud in the predetermined real space synthesized in the order of the transmission timings of each sensor device 1 by the synthesis unit 22.

[0049] As described above, the embodiments of the present invention have been described with reference to the drawings, but the present invention is not limited to the illustrated embodiments. Various modifications and variations can be made within the same scope or equivalent scope as the present invention with respect to the illustrated embodiments.

[0050] 1... Sensor device 11... Sensor unit (LiDAR) 12... Transmission unit 2... Management device 21... Reception unit 22... Synthesis unit 23... Processing unit 24... Scheduler unit

Claims

1. An image sensor data processing system comprising: a plurality of sensor devices arranged on the environment side of a real space, and a management device connected to each sensor device in a communication-enabled manner, wherein each sensor device includes a sensor unit that sequentially acquires image sensor data consisting of point clouds in a predetermined real space by scanning a predetermined real space including dynamic objects, and a transmission unit that transmits frames of image sensor data consisting of point clouds in a predetermined real space acquired sequentially by the sensor unit to the management device at a predetermined transmission period, wherein the management device includes a receiving unit that receives each frame of image sensor data consisting of point clouds in a predetermined real space transmitted from each sensor device, a combining unit that combines each frame of image sensor data consisting of point clouds in a predetermined real space received by the receiving unit in the order of the transmission timing of each sensor device, a processing unit that performs predetermined processing on the frames of image sensor data consisting of point clouds in a predetermined real space that have been combined by the combining unit in the order of the transmission timing of each sensor device, and a scheduler unit that controls the transmission timing of frames of image sensor data consisting of point clouds in a predetermined real space for each sensor device.

2. The image sensor data control system according to claim 1, wherein the scheduler unit controls the transmission timing of frames of image sensor data consisting of a point cloud in a predetermined real space to be shifted by T / N for each sensor device, where T is the transmission period of frames in each sensor device and N is the number of sensor devices.

3. The image sensor data control system according to claim 1, wherein the processing unit transmits each frame of image sensor data consisting of a point cloud in a predetermined real space, which has been synthesized by the synthesis unit in the order of the transmission timing of each sensor device, to a subsequent digital twin system or AI system.

4. The image sensor data control system according to claim 1, wherein the processing unit performs a process for detecting and / or tracking a dynamic object.

5. A management device in an image sensor data control system according to claim 1, comprising a plurality of sensor devices arranged on the real-space environment side and a management device connected to each sensor device in a communication-enabled manner, the management device comprising: a receiving unit that receives each frame of image sensor data consisting of a point cloud in a predetermined real space transmitted from each sensor device; a combining unit that combines each frame of image sensor data consisting of a point cloud in a predetermined real space received by the receiving unit in the order of transmission timing of each sensor device; a processing unit that performs a predetermined process on the frame of image sensor data consisting of a point cloud in a predetermined real space that has been combined by the combining unit in the order of transmission timing of each sensor device; and a scheduler unit that controls the transmission timing of the frame of image sensor data consisting of a point cloud in a predetermined real space for each sensor device.

6. A computer program for use in an image sensor data control system according to claim 1, comprising a plurality of sensor devices arranged on the environment side of a real space, and a management device connected to each sensor device in a communication-enabled manner, wherein the sensor devices are configured to function as: a sensor unit that sequentially acquires image sensor data consisting of point clouds in a predetermined real space by scanning a predetermined real space including dynamic objects; a transmission unit that transmits frames of image sensor data consisting of point clouds in a predetermined real space acquired sequentially by the sensor unit to the management device at a predetermined transmission period; and the management device is configured to function as: a receiving unit that receives each frame of image sensor data consisting of point clouds in a predetermined real space transmitted from each sensor device; a combining unit that combines each frame of image sensor data consisting of point clouds in a predetermined real space received by the receiving unit in the order of the transmission timings of each sensor device; a processing unit that performs predetermined processing on the frames of image sensor data consisting of point clouds in a predetermined real space that have been combined by the combining unit in the order of the transmission timings of each sensor device; and a scheduler unit that controls the transmission timing of frames of image sensor data consisting of point clouds in a predetermined real space for each sensor device.

7. A computer program used in the management device in the image sensor data control system according to claim 1, comprising a plurality of sensor devices arranged on the real-space environment side and a management device connected to each sensor device in a communication-enabled manner, wherein the management device functions as: a receiving unit that receives each frame of image sensor data consisting of a point cloud in a predetermined real space transmitted from each sensor device; a combining unit that combines each frame of image sensor data consisting of a point cloud in a predetermined real space received by the receiving unit in the order of transmission timing of each sensor device; a processing unit that performs predetermined processing on the frame of image sensor data consisting of a point cloud in a predetermined real space that has been combined by the combining unit in the order of transmission timing of each sensor device; and a scheduler unit that controls the transmission timing of the frame of image sensor data consisting of a point cloud in a predetermined real space for each sensor device.

Images