Method and computer system for controlling movable object on basis of sensing data from virtual environment corresponding to space

Abstract

Provided is a control method for controlling a movable object in a space, which is performed by the movable object or a control system for controlling the movable object. The movable object control method comprises: obtaining virtual sensing data for a simulated virtual environment corresponding to a space; and controlling an operation of the movable object in the space on the basis of the virtual sensing data.

Description

Method for controlling a moving object based on sensing data from a virtual environment corresponding to space and a computer system

[0001] The following description relates to a control method for controlling a moving object within space and a computer system for controlling a moving object.

[0002] Mobile devices, including robots, are used to provide various services in indoor and outdoor spaces. These robots include autonomous robots configured to navigate autonomously within a space to perform specific tasks or provide services.

[0003] The operation of a robot is controlled based on sensing data obtained by sensing the actual space in which the robot travels through sensors. For example, the robot uses sensors such as cameras or distance sensors to recognize objects within the space, such as obstacles, and is controlled to avoid the recognized objects or interact with them.

[0004] However, controlling a robot solely based on sensing data directly acquired by such sensors is inefficient, considering the need for the robot to be equipped with multiple or high-performance sensors to accurately sense the space, and the possibility that sensing may become inaccurate depending on environmental conditions of the space. Furthermore, it is inefficient in that implementing areas to restrict the robot's movement within the space, or providing services to offer additional information, requires altering the design of the actual space or modifying the robot's control algorithms.

[0005] Therefore, technology is required to accurately control the robot's operation in various situations without changing the robot's system configuration or algorithms, or the design of the space.

[0006] Korean Patent Publication No. 10-2024-0068268 discloses a robot movement control method and a control server implementing the same.

[0007] The information described above is for illustrative purposes only and may include content that does not constitute part of the prior art and may not include what the prior art would present to a person skilled in the art.

[0008] A method for controlling a moving object can be provided, comprising acquiring virtual sensing data for a simulated virtual environment corresponding to a space and controlling the movement of the moving object within the space based on the virtual sensing data.

[0009] By fusing virtual sensing data about a virtual environment with actual sensing data acquired about a real space and utilizing it for the control of a mobile body, a method for controlling a mobile body can be provided that selectively utilizes virtual sensing data according to environmental conditions in which the robot is driving or the type of sensing data to control the movement of the mobile body within the space.

[0010] In one aspect, a method for controlling a moving body in a space, performed by a moving body or a control system controlling said moving body, is provided, comprising: a step of acquiring virtual sensing data for a simulated virtual environment corresponding to said space; and a step of controlling the operation of said moving body in said space based on said virtual sensing data.

[0011] The virtual environment is a digital twin or a three-dimensional model that simulates the space, and the virtual sensing data may be acquired by at least one virtual sensor placed within the virtual environment.

[0012] The virtual environment includes at least one virtual object that does not actually exist within the space, and the virtual sensing data is acquired based on sensing by the virtual sensor regarding the virtual object, and the controlling step can control the operation of the moving body within the space to avoid the virtual object or interact with the virtual object based on the virtual sensing data.

[0013] The above-described moving object control method may further include the step of acquiring actual sensing data obtained by sensing the space by at least one sensor of the moving object.

[0014] The sensor and the virtual sensor may be sensors of different types, or the sensor and the virtual sensor may be sensors of the same different type but with different detection ranges.

[0015] The above method for controlling a moving object may include a step of controlling the operation, wherein, according to a predetermined condition, the method may include a step of controlling the operation of the moving object within the space using at least one of the actual sensing data or the virtual sensing data.

[0016] According to the above predetermined conditions, the step of controlling the operation of the moving body within the space using at least one of the actual sensing data and the virtual sensing data may control the operation of the moving body within the space based on having a larger or smaller value among the actual sensing data and the virtual sensing data when the actual sensing data and the virtual sensing data are of the same type.

[0017] The above actual sensing data is data sensed by a camera as the actual sensor, and the above virtual sensing data is data sensed by a virtual distance sensor, and the step of controlling the operation of the moving body within the space using at least one of the actual sensing data and the above virtual sensing data according to the above predetermined conditions can control the operation including driving of the moving body within the space using both the actual sensing data and the above virtual sensing data.

[0018] According to the above predetermined conditions, the step of controlling the operation of the mobile body within the space using at least one of the actual sensing data and the virtual sensing data may control the operation of the mobile body within the space using the virtual sensing data instead of the actual sensing data when the mobile body is at a specific location within the space or enters a specific area.

[0019] According to the above predetermined conditions, the step of controlling the operation of the moving body within the space using at least one of the actual sensing data and the virtual sensing data may include the step of masking the data among the actual sensing data and the virtual sensing data that is not used to control the operation of the moving body within the space, or the step of overriding the unused data with the data among the actual sensing data and the virtual sensing data that is used to control the operation of the moving body within the space.

[0020] The above-mentioned controlling step may include a step of providing information about the virtual object or the virtual environment through the moving body based on the virtual sensing data.

[0021] The above virtual object can correspond to an AR (Augmented Reality) object displayed in an AR view corresponding to the above virtual environment.

[0022] The above virtual object may include a virtual wall that blocks the movement of the above-mentioned moving body.

[0023] The virtual sensing data includes data representing a repulsive force toward the virtual object, and the controlling step can control the moving body with a force moving away from the virtual object based on the virtual sensing data.

[0024] The virtual sensing data includes data representing a virtual resistance for decelerating the moving body within the space, and the controlling step can control the moving body to decelerate based on the virtual sensing data.

[0025] The above-mentioned acquisition step acquires updated virtual sensing data for the updated virtual environment as the virtual environment is updated, and the above-mentioned control step can control the operation of the mobile body within the space based on the updated virtual sensing data.

[0026] In another aspect, a computer system comprising a moving body or a control system for controlling said moving body is provided, the computer system comprising at least one processor implemented to execute computer-readable commands, wherein the at least one processor controls the moving body in a space, acquires virtual sensing data for a simulated virtual environment corresponding to said space, and controls the operation of said moving body in said space based on said virtual sensing data.

[0027] By fusing virtual sensing data for a virtual environment with real sensing data acquired for a real space and utilizing it for the control of a moving object, the accuracy of controlling the moving object within the space can be improved even if the moving object includes only a minimum number of sensors or does not include high-performance sensors. In addition, depending on the environmental conditions of the space, the moving object can be effectively controlled even in areas where it is impossible to acquire sensing data by the moving object's sensors.

[0028] By reflecting changes, such as adding and / or changing areas where the movement of a mobile body is restricted within the space where the mobile body operates, as changes to the virtual space and controlling the mobile body based on virtual sensing data for this virtual environment, the mobile body can be controlled in the changed environment without changing the control algorithm of the mobile body or changing the design of the space.

[0029] FIG. 1 illustrates a method for controlling the operation of a moving body within space based on virtual sensing data for a virtual environment corresponding to space, according to one embodiment.

[0030] FIG. 2 is a block diagram showing a moving body and its control system according to one embodiment.

[0031] FIGS. 3 and FIGS. 4 are block diagrams illustrating a control system for controlling a moving body according to one embodiment.

[0032] FIG. 5 is a flowchart illustrating a method for controlling the operation of a moving body within space based on virtual sensing data for a virtual environment corresponding to space, according to one embodiment.

[0033] FIG. 6 is a flowchart illustrating a method for controlling a moving object by fusing virtual sensing data for a virtual environment and actual sensing data for a space according to one example.

[0034] FIG. 7 illustrates a method for acquiring virtual sensing data by sensing by virtual sensor(s) in a simulated virtual environment corresponding to a space, according to one example.

[0035] FIGS. 8 and 9 illustrate a method for controlling the movement of a moving body within space based on virtual sensing data for a virtual environment and / or real sensing data for space, according to one example.

[0036] FIG. 10 illustrates a method for providing information about a virtual object or a virtual environment through a moving body based on virtual sensing data about a virtual environment according to one example.

[0037] FIG. 11 illustrates a method in which a moving body is controlled to decelerate in space or repel a specific virtual object by virtual sensing data according to one example.

[0038] Hereinafter, embodiments will be described in detail with reference to the attached drawings.

[0039]

[0040] FIG. 1 illustrates a method for controlling the operation of a moving body within space based on virtual sensing data for a virtual environment corresponding to space, according to one embodiment.

[0041] The illustrated mobile body (100) may be configured to travel within a space (50). The mobile body (100) may be a robot configured to provide services in an indoor or outdoor space, such as a building, for example. Meanwhile, the mobile body (100) may be any type of means of transportation or mobile body that travels (or drives autonomously) within the indoor or outdoor space (50), such as a vehicle or a trolley. As illustrated, such a mobile body (100) may be configured to move within the space (50) under control by a (robot) control system (120).

[0042] The moving body (100) can autonomously drive through space according to control by a control system (120), and can be configured to drive through space by manual control by a user if necessary.

[0043] The space (50) in which the mobile body (100) moves (or travels) is a place where the mobile body (100) provides services, and may represent, for example, an indoor and / or outdoor space included in a building. The building may include a space where multiple personnel (hereinafter referred to as users) work or reside, and may include, for example, multiple partitioned spaces. The space (50) may represent a part of the building (a specific floor or a partial space within that floor).

[0044] If the mobile body (100) is a service robot, the mobile body (100) may be configured to provide service in at least one layer included in the space (50). Although only one mobile body (100) is shown in FIG. 1, there may be multiple mobile bodies (100) deployed and operating within the space (50). Within the space (50), each of the mobile bodies (100) may move to provide service to an appropriate location or appropriate user within the space.

[0045] The services provided by the mobile body (100) may include, for example, at least one of a delivery service, a delivery service for beverages (such as coffee) ordered, a cleaning service, and other information / content provision services.

[0046] The mobile body (100) may be configured to provide services to a specific user at a specific location in a space (50) by moving by autonomous driving or by moving through manual control, and the movement by (each) autonomous driving and the provision of services of the mobile body (100) may be controlled by a control system (120).

[0047] The moving body (100) can move within the space (50) to achieve a desired purpose, such as providing a service, and can be controlled within the space (50) to move toward a desired destination along a predetermined path.

[0048] Additionally, when the moving body (100) operates within the space (50), it can sense its surroundings within the space (50) using at least one sensor, and its operation can be controlled based on the sensing data obtained from such sensing. Based on the sensing data, the driving unit of the moving body (100) can be controlled, and thus, the moving body (100) can be configured to drive while avoiding an object (30) including an obstacle within the space (50), or to interact with the object (30).

[0049] Meanwhile, in the embodiment, the movement of the moving body (100) can also be controlled based on virtual sensing data for a virtual environment (55) rather than a space (50). Below, to distinguish it from the virtual environment (55), the space (50) may also be referred to as a real-world space (50).

[0050] In other words, the driving unit of the moving body (100) can be controlled based on virtual sensing data for a virtual environment (55) constructed separately from the real world space (50), rather than actual sensing data obtained by sensing the real world space (50) by a sensor, and thus the moving body (100) can be configured to drive while avoiding a virtual object (32) including an obstacle in the virtual environment (55) or to interact with the virtual object (32).

[0051] The virtual environment (55) is constructed by a virtual environment construction system (150) and may include a virtual space corresponding to a real-world environment (50). That is to say, the virtual environment (55) may be a simulated virtual space corresponding to a real-world environment (50). For this virtual environment (550), virtual sensing data may be obtained through sensing by an agent (or a virtual sensor included by the agent) corresponding to a moving body (100). This virtual sensing data may be processed into a control signal for controlling the moving body (100) (i.e., the driving unit of the moving body (100)) as digital data, just like actual sensing data.

[0052] In this way, in the embodiment, the moving body (100) can recognize virtual sensing data for the virtual environment (50) as actual sensing data, and the operation of the moving body (100) can be controlled in the same way as by actual sensing data.

[0053] As illustrated in FIG. 1, in the embodiment, the moving body (100) can not only avoid or interact with real-world objects (30-1 to 30-3) based on real sensing data for a real-world space (50), but also avoid or interact with virtual objects (32-1 to 32-3) based on virtual sensing data for a virtual environment (55).

[0054] The virtual object (32) may be a virtual feature such as an obstacle (32-2), a person (32-1), or a virtual wall (32-3), just like a real-world object (32). Alternatively, the virtual object (32) may be defined as a certain area within a virtual environment (55) that restricts the movement of the moving body (100).

[0055] Meanwhile, if the virtual environment (55) is implemented in the same way as the real-world space (50), that is, by reflecting all objects (50) within the real-world space (50), the mobile body (100) can drive within the space (50) using only virtual sensing data, even without including sensors. As such, in the embodiment, depending on how the virtual environment (55) is implemented, the mobile body (100) can be implemented in a sensor-less form, or the mobile body (100) can be implemented to provide various extended functions or services through the virtual environment (55) which is not limited to the physical implementation of the space (50). Providing extended functions or services to such a mobile body (100) can be made possible simply by updating the virtual environment (55) and acquiring the updated virtual sensing data, without the need to change the physical configuration of the mobile body (100) or the control algorithm, or physically change the real-world space (50).

[0056] A specific embodiment in which a moving body (100) is controlled based on virtual sensing data for a virtual environment (55) will be described in more detail with reference to FIGS. 2 to 11, which will be described later.

[0057]

[0058] Below, the mobile body (100) of the embodiment is described in more detail with reference to FIG. 2. The mobile body (100) may be a service robot for providing services within a space as described above, an autonomous driving robot for other purposes, or a part of such a robot (i.e., a module constituting a part of the robot). That is, the mobile body (100) may be implemented to be capable of autonomous driving, but also capable of manual control according to the user's needs.

[0059] FIG. 2 is a block diagram showing a moving body and its control system according to one embodiment. In addition, FIG. 2 separately illustrates a virtual space construction system (150) for constructing a virtual space (55).

[0060] As illustrated in FIG. 2, the moving body (100) is a physical device and may include a control unit (104), a driving unit (108), a sensor unit (106), and a communication unit (102). At least some of the illustrated components (102 to 108) may be implemented in a form included in a computer system included in the moving body (100).

[0061] The control unit (104) may be a physical processor embedded in the mobile body (100) (or computer system) and may include a path planning processing module, a mapping processing module, a driving control module, a localization processing module, a data processing module, and a service processing module as components to support autonomous driving of the mobile body (100), although not separately illustrated. In this case, the path planning processing module, the mapping processing module, and the localization processing module may be optionally included in the control unit (104) according to the embodiment to enable indoor autonomous driving of the mobile body (100) even when communication with the control system (120) is not established. At least a portion of the control unit (104) may be included in the autonomous driving module provided in the mobile body (100).

[0062] The communication unit (102) may be configured for the mobile body (100) to communicate with another device (another robot / robot or control system (120), etc.). That is to say, the communication unit (102) may be a hardware module such as an antenna, data bus, network interface card, network interface chip, and networking interface port of the mobile body (100), or a software module such as a network device driver or a networking program, which transmits / receives data and / or information to / from another device.

[0063] The drive unit (108) is a configuration that controls the movement of the moving body (100) and enables movement, and may include equipment for performing this. The drive unit (108) may be a device that drives the moving body (100) for autonomous driving of the moving body (100) based on sensing data from the sensor unit (106). The drive unit (108) may include at least one actuator, motor, motor driver, and other mechanism as an electronic and / or mechanical device that operates for driving the moving body (100). For example, the moving body (100) may drive through space (50) by operating a wheel by a motor. Additionally, the drive unit (108) may include a battery for operating the moving body (100). The drive unit (108) may be a configuration for driving the moving body (100), for example, a configuration included in a driving platform provided at the bottom of the moving body (100).

[0064] The sensor unit (106) may be configured to collect sensing data required for autonomous driving and service provision of the mobile body (100). The sensor unit (106) may include at least one sensor. The sensor unit (106) may be configured to include a computing module that is included in the aforementioned autonomous driving module and corresponds to at least a part of the aforementioned control unit (104).

[0065] The sensor unit (106) may not include expensive sensing equipment and may only include sensors such as low-cost ultrasonic sensors and / or low-cost cameras. The sensor unit (106) may include sensors for identifying other robots or people in front and / or behind, obstacles or dangerous areas in the surroundings (such as the floor in the space). For example, the sensor unit (106) may include at least one of a vision sensor (such as an optical sensor for vision recognition) and a distance sensor, such as a depth camera, an infrared camera or sensor, an RGB camera, etc., as sensors. Through these sensors, other robots, people, and other features in the surroundings can be recognized. These sensors may be appropriately selected to have high resolution and a rich amount of information regarding the surrounding environment. The driving unit (108) can be controlled based on actual sensing data obtained through sensing by the sensors included in the sensor unit (106), and thus, the movement of the moving body (100) can be controlled.

[0066] The mobile body (100) can operate a drive unit (108) and a robot arm (not illustrated), etc., to provide a service by receiving a command received through the robot control system (120) via the communication unit (102) or via the communication unit (102) and a data processing module through the service processing module of the aforementioned control unit (104). The service processing module can transmit a drive command for the service to be provided to the drive control module, and the drive control module can control the configuration including the drive unit (108) and the robot arm of the mobile body (100) according to the drive command so that the service can be provided. Additionally, although not illustrated, the mobile body (100) may further include an output interface, such as a speaker and / or a display, for outputting information / content necessary for providing the service.

[0067] Meanwhile, the mobile body (100) may be distinct from a mapping robot (i.e., a mapping robot) used to generate an indoor map within a space. Since the mobile body (100) does not include expensive sensing equipment, it can perform indoor autonomous driving using the output values ​​of sensors such as low-cost ultrasonic sensors and / or low-cost cameras. Meanwhile, if the mobile body (100) has previously performed indoor autonomous driving through communication with the control system (120), it may be possible to perform more accurate indoor autonomous driving while using low-cost sensors by further utilizing mapping data, etc., included in the path data (e.g., data about the path) previously received from the control system (120). However, depending on the embodiment, the mobile body (100) may also serve as the mapping robot.

[0068] As described above, if the moving body (100) only provides sensing data for controlling the moving body (100) to the control system (120) and the algorithm for controlling the moving body (100) is executed in the control system (120), the moving body (100) may correspond to a brainless robot.

[0069] Meanwhile, as described above, the moving body (100) may be configured so that the driving unit (108) can be controlled based on virtual sensing data for a virtual environment (55) rather than actual sensing data. According to an embodiment, the moving body (100) may be implemented in a sensorless form that does not include a sensor unit (106) at all.

[0070] The control unit (104) can control the driving unit (108) based on actual sensing data and / or virtual sensing data, and can control the driving unit (108) by selectively using actual sensing data or virtual sensing data according to a predetermined condition (or standard), or by fusing actual sensing data and virtual sensing data. In this regard, an embodiment will be described in more detail with reference to the drawings to be described later.

[0071] Each of the moving bodies (100) may have different sizes and shapes depending on the type of machine or the service provided, and is not limited to the shape shown in the drawing.

[0072] The configuration and operation of the control system (120) controlling the moving body (100) will be described in more detail with reference to FIG. 3 and FIG. 4, which will be described later.

[0073] FIGS. 3 and FIGS. 4 are block diagrams illustrating a control system for controlling a moving body according to one embodiment.

[0074] The control system (120) may be a device for controlling the operation of the components of the aforementioned mobile body (100) for movement (i.e., autonomous driving) within the space of the aforementioned mobile body (100) and for providing services within the space. For example, the control system (120) may transmit a control signal for controlling the aforementioned drive unit (108) to the control unit (104).

[0075] The control system (120) can control the movement of each of the plurality of mobile bodies (100) and the provision of services for each of the mobile bodies (100). The control system (120) can set a path to a destination for the mobile body (100) to provide services through communication with the mobile body (100), and can transmit information regarding such a path to the mobile body (100). The mobile body (100) can drive autonomously according to the received information regarding the path.

[0076] The control system (120) may include at least one computing device. The control system (120) may be a robot control system or part thereof for controlling a mobile body (100) which is a robot.

[0077] The control system (120) may be a device that sets a path for driving the mobile body (100) and controls the movement of the mobile body (100) as described above. The control system (120) may include at least one computing device and may be implemented as a server located within or outside the space.

[0078] As illustrated, the control system (120) may include a memory (330), a processor (320), a communication unit (310), and an input / output interface (340).

[0079] The memory (330) is a computer-readable recording medium and may include a non-perishable mass storage device such as RAM (random access memory), ROM (read only memory), and a disk drive. Here, the ROM and the non-perishable mass storage device may be included as separate permanent storage devices separated from the memory (330). Additionally, an operating system and at least one program code may be stored in the memory (330). These software components may be loaded from a computer-readable recording medium separate from the memory (330). This separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, disk, tape, DVD / CD-ROM drive, or memory card. In another embodiment, the software components may be loaded into the memory (330) through a communication unit (310) rather than a computer-readable recording medium.

[0080] The processor (320) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (320) by memory (330) or a communication unit (310). For example, the processor (320) may be configured to execute instructions received according to program code loaded in memory (330). Such a processor (320) may include configurations (410 to 440) as illustrated in FIG. 4. The processor (320) may, for example, generate a control signal (command) for controlling the aforementioned drive unit (108) (wheel, actuator, motor, etc.) and command the control unit (104) of the moving body (100) to control the drive unit (108).

[0081] Each of the components (410 to 440) of the processor (320) may be a software and / or hardware module as part of the processor (320) and may represent a function (function block) implemented by the processor. The components (410 to 440) of the processor (320) will be described later with reference to FIG. 4.

[0082] The communication unit (310) may be a configuration for the control system (120) to communicate with another device (such as a mobile body (100) or another server). That is to say, the communication unit (310) may be a hardware module such as an antenna, data bus, network interface card, network interface chip, and networking interface port of the control system (120) that transmits / receives data and / or information to / from another device, or a software module such as a network device driver or a networking program.

[0083] The input / output interface (340) may be a means for interfacing with an input device such as a keyboard or mouse and an output device such as a display or speaker.

[0084] Additionally, in other embodiments, the control system (120) may include more components than the illustrated components.

[0085] With reference to FIG. 4, the components (410 to 440) of the processor (320) will be described in more detail. As illustrated, the processor (320) may include a map generation module (410), a localization processing module (420), a path planning processing module (430), and a service operation module (440). The components included in the processor (320) may be representations of different functions performed by at least one processor included in the processor (320) according to control instructions according to the code of an operating system or the code of at least one computer program.

[0086] The map generation module (410) may be a component for a mapping robot (not shown) autonomously driving within a space to generate an indoor map of a target facility (e.g., using sensing data generated about the interior of the space).

[0087] At this time, the localization processing module (420) can determine the location of the mobile body (100) inside the target facility using the sensing data received from the mobile body (100) through the network and the indoor map of the target facility generated through the map generation module (410).

[0088] The path planning processing module (430) can generate a control signal for controlling the indoor autonomous driving of the vehicle (100) using the actual sensing data received from the vehicle (100) described above and / or the virtual sensing data obtained from the virtual environment construction system (150) and the generated indoor map. For example, the path planning processing module (430) can generate a path (i.e., path data) of the vehicle (100). The generated path data can be set for the vehicle (100) for driving the vehicle (100) along the path. The control system (120) can transmit information regarding the generated path to the vehicle (100) via a network. For example, the information regarding the path may include information indicating the current location of the vehicle (100), information for mapping the current location to the indoor map, and path planning information. The information regarding the path may include information regarding the destination of the space where the vehicle (100) must drive to a predetermined location within the space or to provide services to a predetermined user. The path planning processing module (430) can set a path for the mobile body (100). The control system (120) can control the movement of the mobile body (100) so that the mobile body (100) moves according to this set path (i.e., along the set path). Additionally, the control signal may include a signal transmitted to the mobile body (100) to control the driving unit (108) of the mobile body (100).

[0089] The service operation module (440) may include a function for controlling the services provided by the mobile body (100) within the space. For example, the control system (120) or the service provider operating the space may provide an Integrated Development Environment (IDE) for the services provided by the control system (120) (e.g., cloud services) to the user or creator of the mobile body (100). At this time, the user or creator of the mobile body (100) may create software to control the services provided by the mobile body (100) within the space through the IDE and register it with the control system (120). In this case, the service operation module (440) may control the services provided by the mobile body (100) using the software registered in association with the mobile body (100). As a specific example, assuming that a mobile body (100) provides a service of delivering a payload requested by a user to the user's location, the control system (120) can transmit related commands to the mobile body (100) so that the mobile body (100) not only controls the indoor autonomous driving of the mobile body (100) to move to the user's location, but also controls a robot arm, etc., when it arrives at the destination location to deliver an object to the user and output a user response voice, thereby enabling the mobile body (100) to provide a series of services.

[0090] The control system (120) may be a server as a computer system for controlling the mobile body (100). The control system (120) may be a cloud server as a server located outside a space or building. Alternatively, depending on the embodiment, the control system (120) may be located inside a space or building.

[0091] The description of the technical features described above with reference to FIGS. 1 and 2 can be applied as is to FIGS. 3 and 4, so redundant descriptions are omitted.

[0092] Meanwhile, the virtual environment construction system (150) may be a server as a computer system for constructing a virtual environment (55) corresponding to a real-world space (50). The virtual environment construction system (150) may also be a cloud server as a server placed outside a space or building. Alternatively, depending on the embodiment, the virtual environment construction system (150) may be placed inside a space or building.

[0093] The virtual environment construction system (150) can construct a digital twin or a three-dimensional model that simulates a real-world space (50), and the virtual environment (55) may be at least part of such constructed digital twin or three-dimensional model. Since the description of the configurations of the control system (120) described above can be similarly applied to the configurations of the virtual environment construction system (150), redundant descriptions are omitted.

[0094]

[0095] In the detailed description to be provided below, the operation performed by the components of the control system (120) or the moving body (100) (e.g., control unit (104), processor (320), etc.) may be described as an operation performed by the control system (120) or the moving body (100) for convenience of explanation.

[0096] Additionally, operations or steps for acquiring virtual or actual sensing data and controlling the moving body (100), which will be described later with reference to FIGS. 5 to 11, may be performed by the moving body (100), and according to the embodiment, at least some of these may be performed by the moving body (100) in accordance with a control signal from the control system (120). Below, embodiments are described with a focus on operations or steps being performed by a computer system (100, 120) constituting the moving body (100) or the control system (120), and redundant descriptions of embodiments involving the control system (120) or the moving body (100) may be omitted.

[0097]

[0098] FIG. 5 is a flowchart illustrating a method for controlling the operation of a moving body within space based on virtual sensing data for a virtual environment corresponding to space, according to one embodiment.

[0099] Below, a method of controlling a moving body (100) in which each step is performed by a computer system (100, 120) constituting the moving body (100) or the control system (120) is described in more detail.

[0100] In step (505), the computer system (100, 120) can acquire real sensing data that senses real-world space (50) by at least one sensor included in the sensor unit (106) of the moving body (100). Such a sensor may include, for example, a distance sensor that detects distance, a range sensor that detects an area of ​​a certain range, or a ToF sensor. Additionally, the sensor may include an IMU (Inertial Measurement Unit). The IMU may include an accelerometer, a gyroscope, etc. Additionally, depending on the type of the moving body (100), such a sensor may include at least one camera or / additionally include a LiDAR. The computer system (100, 120) can recognize an obstacle or danger area within real-world space (50) based on the sensing data from such a sensor, and the movement of the moving body (100) can be appropriately controlled based on the distance to the recognized obstacle or danger area.

[0101] For reference, if the moving body (100) is implemented in a sensorless form controlled only by virtual sensor data, step (505) may not be performed.

[0102] In step (510), the computer system (100, 120) can acquire virtual sensing data for a simulated virtual environment (55) corresponding to the space (50). The virtual environment (55) is constructed by a virtual environment construction system (150) and may be at least part of a digital twin or a three-dimensional model that simulates the real-world space (50). This virtual environment (55) may be a simulated space of the real-world space (50) and may be implemented to simulate the movement of the moving body (100) and the sensing by the sensor. The position of the moving body (100) in the virtual environment (55) and the real-world space (50) may be synchronized with each other. Thus, the computer system (100, 120) can acquire virtual sensing data for the virtual environment (55) corresponding to the position when the moving body (100) is at a position in the real-world space (50).

[0103] For example, a virtual environment construction system (150) can simulate the operation of an agent (or a virtual sensor included in the agent) corresponding to a moving body (100) with respect to a virtual environment (55), and accordingly, can acquire virtual sensing data. The virtual environment construction system (150) can transmit the acquired virtual sensing data to a computer system (100, 120), and the virtual sensing data acquired by the computer system (100, 120) can be used to control the operation of the moving body (100).

[0104] At least one virtual sensor may be placed within the virtual environment (55), and virtual sensing data may be obtained by such virtual sensor. For example, the virtual sensor may include a virtual LiDAR, a virtual distance sensor, a virtual camera, a virtual IMU, etc.

[0105] The virtual environment (55) is a virtual space corresponding to the real world space (50), but it may be constructed separately from the real world space (50). Accordingly, the virtual environment (55) may include at least one virtual object (32) that does not actually exist within the real world space (50). Virtual sensing data may be obtained through (virtual) sensing by a virtual sensor placed in the virtual environment (55) for such virtual object (32).

[0106] Virtual sensing data may be injected into the moving body (100) as data acquired by a virtual sensor, rather than being acquired by sensing by a real sensor like the actual sensing data in step (505). However, virtual sensing data can also be used to control the operation of the moving body (100), that is, the operation of the driving unit (108), just like actual sensing data.

[0107] When comparing the sensor actually included in the moving body (100) with the virtual sensor, the actual sensor and the virtual sensor may be of different types of sensors. That is, the virtual sensor may be a sensor for acquiring sensing data of a different type than the sensor included in the moving body (100). Thus, the operation of the moving body (100) of the embodiment can be controlled based on virtual sensing data from a type of sensor that the moving body (100) does not include. For example, even if the moving body (100) does not include a distance sensor, the moving body (100) can operate in the real world space (50) based on virtual sensing data from a virtual distance sensor.

[0108] Alternatively, the actual sensor and the virtual sensor may be sensors of the same different type but with different detection ranges. That is, the virtual sensor may be a sensor for acquiring sensing data of the same type as the sensor included in the moving body (100), but the amount or range of data acquired may be different. For example, the virtual sensor may be a sensor with higher performance or a wider detection range than the sensor included in the moving body (100). Thus, the operation of the moving body (100) of the embodiment can be controlled as if it were equipped with a sensor with higher performance than the sensor actually mounted. For example, the moving body (100) can operate appropriately based on virtual sensing data from the virtual distance sensor even in shaded areas where acquiring sensing data is impossible or inaccurate due to the mounted distance sensor. For example, even if the moving body (100) includes a lidar capable of detecting a range of 180 degrees, the detection range of the moving body (100) can be expanded by injecting virtual sensing data from a virtual lidar capable of detecting a range of 360 degrees into the moving body (100). In this way, virtual sensing data can be utilized to further expand actual sensing data.

[0109] In this way, virtual sensing data can be used to control the operation of a moving body (100) by replacing or assisting actual sensing data.

[0110] In step (520), the computer system (100, 120) can control the movement of the moving body (100) in the real-world space (50) using at least one of the acquired real sensing data or virtual sensing data. For example, the moving body (100) can drive in the real-world space (50), avoid object(s) (30, 32) in the space (50) or virtual environment (55), or interact with object(s) (30, 32) based on the real sensing data and / or virtual sensing data.

[0111] As described above, the computer system (100, 120) can control the movement of the moving body (50) in the real-world space (50) based on the virtual sensing data obtained in step (510). At this time, the computer system (100, 120) can control the movement of the moving body (100) in the space to avoid the virtual object (32) or to interact with the virtual object (32) based on the virtual sensing data.

[0112] Meanwhile, the computer system (100, 120) can control the operation of the moving body (100) in the real world space (50) by using at least one of actual sensing data or virtual sensing data according to a predetermined condition. Thus, the moving body (100) can control the driving unit (108) by selectively using actual sensing data or virtual sensing data according to a predetermined condition, or by fusing actual sensing data and virtual sensing data.

[0113] 1) For example, if the actual sensing data and the virtual sensing data are of the same type, the computer system (100, 120) can control the operation of the moving body (100) by selectively utilizing the actual sensing data or the virtual sensing data to control the driving unit (108). In this regard, the operation of the moving body (100) within the real-world space (50) can be controlled based on having a larger or smaller value among the acquired actual sensing data and the virtual sensing data. Alternatively, the operation of the moving body (100) within the real-world space (50) can be controlled based on having a value within a certain range among the acquired actual sensing data and the virtual sensing data. For example, if both the actual sensing data and the virtual sensing data are sensing data acquired according to sensing by a distance sensor, the moving body (100) can be controlled based on the smaller value among the overlapping sensing data, that is, the value representing a shorter distance. Thus, the moving body (100) can be controlled to react more sensitively to surrounding objects (30, 32).

[0114] 2) Or, if the actual sensing data and the virtual sensing data are different types of data, the computer system (100, 120) can control the operation of the moving body (100) by utilizing both the actual sensing data and the virtual sensing data to control the driving unit (108).

[0115] For example, if the actual sensing data is data sensed by a camera of the moving body (100) as an actual sensor, and the virtual sensing data is data sensed by a virtual distance sensor, then both the actual sensing data and the virtual sensing data can be used to control the movement of the moving body (100) within the real-world space (50). As a result, the actual sensing data and the virtual sensing data can be utilized complementarily in controlling the movement of the moving body (100), and the control of the movement of the moving body (100) in the real-world space (50) can be made more precise.

[0116] Control of the moving body (100) by selectively utilizing actual sensing data and / or virtual sensing data in 1) and 2) can be performed when it is determined that the moving body (100) is in a specific location or area within the space (50) or virtual environment (55).

[0117] 3) For example, a computer system (100, 120) may control the movement of the moving body (100) within the space using virtual sensing data instead of actual sensing data when the moving body (100) is in a specific location (i.e., a preset location) within the space (50) or enters a specific area (i.e., a preset area), or control the movement of the moving body (100) within the space using actual sensing data instead of virtual sensing data, or control the movement of the moving body (100) within the space using both actual sensing data and virtual sensing data.

[0118] To constitute the above-mentioned predetermined conditions, at least two of the aforementioned 1) to 3) may be combined.

[0119] In this way, control of the mobile body (100) can be performed by selectively utilizing actual sensing data and / or virtual sensing data according to settings by an administrator or user operating the mobile body (100). By doing so, the operation of the mobile body (100) can be further stabilized by utilizing virtual sensing data (ideal sensing data) in areas where sensing within the space (50) becomes weak, taking into account environmental conditions of the real world space (50).

[0120] In this regard, FIGS. 8 and 9 illustrate a method for controlling the movement of a moving body within space based on virtual sensing data for a virtual environment and / or real sensing data for space, according to one example.

[0121] In FIG. 8, an example is provided in which both virtual sensing data and actual sensing data acquired by a moving body (100) are data acquired by a distance sensor. For instance, if actual sensing data obtained by the distance sensor of the moving body (100) at a specific location indicates a distance (S2) from an actual object (820), while the corresponding virtual sensing data indicates a distance (S1) from a virtual object (810), the moving body (100) can be controlled based on the virtual sensing data indicating a closer distance. Here, virtual sensing data can be acquired as the virtual distance sensor detects the virtual object (810) at a location on a virtual environment (55) corresponding to the location of the moving body (100). Thus, the movement of the moving body (100) can be controlled to avoid the virtual object (810) or to interact with the virtual object (810).

[0122] In FIG. 9, an example is shown in which distance sensor data, which is virtual sensing data acquired by a moving body (100), and camera sensing data (vision information), which is actual sensing data, are utilized for the control of the moving body (100). At this time, a real object (910) in the real world space can be recognized by the camera. Meanwhile, a virtual object (920) corresponding to the real object (910) can also be implemented in the virtual environment (55), and as the virtual object (920) is sensed by a virtual distance sensor, virtual sensing data can be acquired.

[0123] The moving body (100) can utilize both the vision information of the actual object (910) recognized by the camera and the distance information to the virtual object (920) based on virtual sensing data to avoid, for example, an actual object (910) which is an obstacle. The operation control of the moving body (100) can be performed in areas where sensing by a distance sensor is difficult, or in cases where it is difficult to recognize the distance to the actual object (910) due to the material or shape characteristics of the actual object (910). At this time, even if there is distance information included in the actual sensing data obtained by the distance sensor of the moving body (100), it may be ignored, and virtual sensing data may be used instead for the control of the moving body (100).

[0124] Thus, even when the moving body (100) is located in a shaded area where sensing by a sensor is difficult, safe and effective control of the moving body (100) can be performed.

[0125] Returning to step (520), we will describe in more detail another example in which the movement of a moving body (100) is controlled based on virtual sensing data.

[0126] As in step (522), the computer system (100, 120) can provide information about a virtual object (32) or a virtual environment (55) through the mobile body (100) based on virtual sensing data. For example, the mobile body (100) may further include an output interface, such as a speaker and / or display, for outputting information and / or content necessary for providing a service, and the information may be output through this output interface. For example, the computer system (100, 120) can recognize a virtual object (32), such as a virtual obstacle (32-2), a virtual wall (32-3), or a virtual person (32-1), based on virtual sensing data, and can output the information through the output interface by interacting with the virtual object (32).

[0127] The virtual object (32) can correspond to an AR (Augmented Reality) object displayed in an AR view that displays a virtual environment (55).

[0128] In this regard, FIG. 10 illustrates a method for providing information about a virtual object or a virtual environment through a moving body based on virtual sensing data about a virtual environment according to one example.

[0129] In the example illustrated in FIG. 10, an embodiment is illustrated in which a moving body (100) includes a display (1000) and an AR view (1010) is displayed through the display. The AR view (1010) includes images captured by a camera of the moving body (100) and may correspond to a real-world space (50) and a virtual environment (55). The AR view (1010) may include at least one AR object (1020). The AR object (1020) may be augmented content that is displayed augmented in the AR view (1010). The AR object (1020) may be configured to provide specific information and content. For example, the AR object (1020) may have specific information and content registered or mapped to it, and may be configured to display content (1030) in the AR view (1010) when the moving body (100) approaches the AR object (1020). The content (1030) may include information about facilities located in the space (50), such as restaurants.

[0130] In an embodiment, based on virtual sensing data, the moving body (100) can recognize a virtual object (32) that does not exist in the real-world space (50) but is placed within a virtual environment (55), and accordingly, can display an AR object (1020) and associated content (1030) within an AR view (1010) through a display (1000).

[0131] In this way, the virtual environment (55) constructed in correspondence with the space (50) is linked with the AR view (1010) and can be used to provide AR objects (1020) and content (1030) through the AR view (1010). That is to say, various virtual objects (32) (entities) implemented within the virtual environment (55) can interact with the moving body (100) and can also be linked with AR objects within the AR view (1010).

[0132] Below, as another embodiment in which the movement of a moving body (100) is controlled based on virtual sensing data, a method for controlling the movement of a moving body (100) when the virtual object (32) is a virtual wall (32-3) is described in more detail.

[0133] A virtual object (32) implemented within a virtual environment (55) may include a virtual wall (32-3) that blocks the movement of a moving body (100). When the moving body (100) collides with the virtual wall (32-3), the moving body (100) can be controlled in a manner similar to colliding with a real wall (30-3). For example, just as a repulsive force is applied to the moving body (100) when it collides with a real wall (30-3), a repulsive force may also be applied to the moving body (100) when it collides with the virtual wall (32-3). This repulsive force may be a repulsive force that causes the virtual wall (32-3) and the moving body (100) to move away from each other. At this time, the virtual sensing data acquired by the moving body (100) may include data indicating the repulsive force against the virtual object (32) (virtual wall (32-3)). The computer system (100, 120) can control the moving body with a force moving away from the virtual object (32) (virtual wall (32-3)) based on virtual sensing data containing data representing such repulsive force. In other words, the moving body (100) can be repulsively controlled.

[0134] In this regard, FIG. 11 illustrates a method in which a moving body is controlled to decelerate in space or repel a specific virtual object by virtual sensing data according to one example.

[0135] In FIG. 11, a virtual object (1125) is illustrated as an example of a virtual object (32). The virtual object (1125) may be a specific area within the aforementioned virtual wall (32-3) or virtual environment (55). When a moving body (100) comes into contact with such a virtual object (1125) or approaches the virtual object (1125) beyond a certain limit, it may acquire virtual sensing data including data indicating a repulsive force toward the virtual object (1125), and thus may be controlled by a force moving away from the virtual object (1125) (repulsion control (1120)). Thus, the moving body (100) may be controlled by considering the contact situation with the elastic virtual object (32) regardless of the structure of the real-world space (50).

[0136] Alternatively, when the moving body (100) comes into contact with the virtual object (1125) or approaches the virtual object (1125) beyond a certain level, it may acquire virtual sensing data including data representing a virtual resistance for decelerating the moving body (100) within the space (50), and thus, the moving body (100) may be controlled to decelerate based on this virtual sensing data (deceleration control (1110)). The resistance may be designed to increase as it gets closer to the virtual object (1125) or / additionally proportional to the speed of the moving body (100).

[0137] In addition, virtual sensing data that generates a virtual flow rate can be configured so that the moving body (100) moves accordingly within a specific area, and virtual sensing data including data representing a repulsive force as described above can be configured so that the moving body (100) moves Brownian motion.

[0138] As described in the embodiments, the moving body (100) can be controlled in various ways based on injected virtual sensing data, which is different from the actual sensing data directly acquired by the moving body (100) through a sensor. The virtual sensing data may be data that defines the effect on the moving body (100) when the moving body (100) travels in a virtual environment (55) constructed by modeling the space (50) in three dimensions. The virtual sensing data may include distance-based sensing data, vision-based sensing data, force-based sensing data, etc., and may be acquired through simulation of the virtual environment (55). The virtual sensing data can be injected into the moving body (100) and used to control the moving body (100) (i.e., the driving unit (108)) just like actual sensing data.

[0139] Meanwhile, virtual sensing data may be updated as the virtual environment (55) is updated. As the virtual environment (55) is updated, the computer system (100, 120) may acquire updated virtual sensing data for the updated virtual environment (55) and control the movement of the moving body (100) within the space based on the updated virtual sensing data. The update of the virtual environment (55) may be, for example, a change (i.e., addition, deletion, movement, etc.) of a virtual object (55) within the virtual environment (55).

[0140] As in step (512), the computer system (100, 120) can identify updates to the virtual environment (55). The computer system (100, 120) can identify updates to the virtual environment (55) in accordance with communication with the virtual environment building system (150). In step (514), the computer system (100, 120) can obtain updated virtual sensing data for the updated virtual environment (55). The mobile body (100) can travel through space (50) while taking into account changes in the updated virtual environment (55) by being controlled based on the updated virtual sensing data.

[0141] Thus, in the embodiment, even when the mobile body (100) frequently changes or reassigns areas where it cannot move within the space (50), this can be immediately reflected by simply updating the virtual environment (55), thereby increasing the efficiency of the operation of the mobile body (100). Furthermore, since the mobile body (100) can be controlled based on virtual sensing data, which is abstract data, complex processes such as changing the system configuration or control algorithm of the mobile body (100) are not required to drive through the space (50) while considering changes in the updated virtual environment (55).

[0142] If the update of the virtual environment (55) is performed to reflect real-time changes in the real-world space (50), and the computer system (100, 120) is configured to acquire the updated virtual sensing data in real-time, the mobile body (100) can operate properly in the real-world space (50) even without having a sensor. In other words, the mobile body (100) can be implemented in a sensorless form.

[0143] The description of the technical features described above with reference to FIGS. 1 to 4 can be applied as is to FIGS. 5 and FIGS. 8 to 11, so redundant descriptions are omitted.

[0144]

[0145] FIG. 6 is a flowchart illustrating a method for controlling a moving object by fusing virtual sensing data for a virtual environment and actual sensing data for a space according to one example.

[0146] Below, with reference to Fig. 6, the acquisition of actual sensor data and virtual sensor data and the fusion of actual sensor data and virtual sensor data are described in more detail.

[0147] As described above with reference to step (505) of FIG. 5, the computer system (100, 120) can acquire real sensing data by sensing the real world space (50) by at least one sensor included in the sensor unit (106) of the moving body (100) (step (610)). Meanwhile, as described above with reference to step (510), the computer system (100, 120) can acquire virtual sensing data for a simulated virtual environment (55) corresponding to the real world space (50) (step (620)). To this end, the virtual environment construction system (150) can first construct a virtual environment (55) that models the real world space (50) in three dimensions, and can place at least one virtual sensor within the virtual environment (55) or drive an agent including virtual sensor(s) corresponding to the moving body (100) to acquire virtual sensing data. Thus, sensor input simulation can be performed on the virtual environment (55) (step (630)). The computer system (100, 120) can obtain virtual sensing data from the virtual environment construction system (150) (step (620)).

[0148] To control the driving unit (108), the acquired actual sensing data or virtual sensing data may be used selectively, or the actual sensing data and virtual sensing data may be fused and used.

[0149] As described in step (520), the computer system (100, 120) can control the operation of the moving body (100) in the real world space (50) using at least one of actual sensing data and virtual sensing data according to a predetermined condition.

[0150] In order to control the operation of the moving body (100) within the real-world space (50), real sensing data and virtual sensing data can be fused (step (630)), and the fused sensor data can be converted into a control signal for controlling the driving unit (108) through processing or manipulation (step (640)). The control of the operation of the moving body (100) may include motors, actuators, speed of the moving body (100), torque control (such as wheels), etc.

[0151] For example, the fusion processing of sensing data and virtual sensing data may include a processing of masking data that is not used to control the operation of the moving body (100) in the real-world space (50) among the actual sensing data and virtual sensing data. Or / additionally, the fusion of sensing data and virtual sensing data may include a processing of overriding the said unused data with data that is used to control the operation of the moving body (100) in the real-world space (50) among the actual sensing data and virtual sensing data.

[0152] The sensing data generated according to this fusion processing may consist only of sensing data used for operation within the real-world space (50) of the moving body (100), and may be converted into a control signal for controlling the driving unit (108) through processing in step (640). That is to say, the fusion processing process described above with reference to step (630) may be a processing process for generating final sensing data used for controlling the moving body (100) based on actual sensing data and virtual sensing data.

[0153] The above fusion processing may be applied to actual sensing data and virtual sensing data using probabilistic methods such as utilizing a Kalman filter or a particle filter, or methods applying logical operations such as AND or OR. In addition, any method for removing redundant sensing data or selecting a more appropriate value among the sensing data, such as the methods 1) to 3) described above with reference to FIG. 5, may be used in the fusion processing.

[0154] Below, with reference to FIG. 7, a method for acquiring virtual sensing data by virtual sensor(s) within a virtual environment (55) is described in more detail.

[0155] In this regard, FIG. 7 illustrates a method for acquiring virtual sensing data by sensing by virtual sensor(s) in a simulated virtual environment corresponding to a space, according to one example.

[0156] In FIG. 7, a virtual environment (700) constructed in correspondence with a real-world space (50) is illustrated. The virtual environment (700) may be a digital twin that models the real-world space (50) in three dimensions. The virtual environment (700) is a virtual space in which the aforementioned sensor input simulation can be performed, and as the sensor input simulation is performed, the virtual environment construction system (150) can acquire virtual sensing data.

[0157] Each of the virtual objects (730-1 to 730-3) placed in the virtual environment (700) may correspond to an object (30) placed in the real world space (50), or may be placed only in the virtual environment (700) and not exist in the real world space (50).

[0158] An agent (710) corresponding to a moving body (100) may be deployed in a virtual environment (700). At this time, the agent (710) may include at least one virtual sensor (720-1, 720-2). While operating (including driving) in the virtual environment (700), the agent (710) may sense virtual objects (730-1 to 730-3) or sense the virtual environment (700), thereby acquiring virtual sensing data. The acquired virtual sensing data may be transmitted to a computer system (100, 120) and used to control the operation of the actual moving body (100). For example, if the virtual sensor (720-1, 720-2) is a LiDAR or a distance sensor, the virtual sensor (720-1, 720-2) may sense virtual objects (730-1 to 730-3) through a raycast. In this way, the virtual sensor (720-1, 720-2) can acquire virtual sensing data by sensing virtual objects (730-1 to 730-3) in a manner similar to that of a real sensor.

[0159] The actual sensing data used by the actual moving body (100) for controlling its operation is information based on the result value measured by the sensor, and may be a digitized value. For example, if the sensor is a laser distance sensor, the distance can be estimated using the difference in the distance (phase) of the laser reflecting back from the transmitting unit to the object, and this data can be used for controlling the operation of the moving body (100) as a digitized value. Since the virtual sensing data of the embodiment is also a digitized value, the moving body (100) cannot distinguish between the actual sensing data and the virtual sensing data. Thus, in the embodiment, the virtual sensing data obtained through simulation of the virtual environment (700) can be utilized for controlling the operation of the moving body (100) just like the actual sensing data.

[0160] In the embodiment, when a moving body (100) operates in a real-world space (50), real sensing data for the real-world space (50) and virtual sensing data for a virtual environment (55) whose location is synchronized with that of the real-world space (50) can be acquired simultaneously according to the positioning in the real-world space (50), and the acquired real sensing data and / or virtual sensing data can be used to control the moving body (100) selectively or together.

[0161] The description of the technical features described above with reference to FIGS. 1 to 5 and FIGS. 8 to 11 can be applied as is to FIGS. 6 and FIGS. 7, so redundant descriptions are omitted.

[0162] The system or device described above may be implemented as a hardware component, a software component, or a combination of a hardware component and a software component. For example, the device and component described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0163] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0164] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0165] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0166] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

1. A control method for controlling a moving body in space, performed by a moving body or a control system controlling said moving body, wherein A step of acquiring virtual sensing data for a simulated virtual environment corresponding to the above space; and A step of controlling the operation of the mobile body within the space based on the virtual sensing data. A method for controlling a moving object, including 2. In Paragraph 1, The above virtual environment is a digital twin or a three-dimensional model that simulates the above space, and A method for controlling a moving object, wherein the virtual sensing data is acquired by at least one virtual sensor placed within the virtual environment.

3. In Paragraph 2, The above virtual environment includes at least one virtual object that does not actually exist within the space, and The above virtual sensing data is acquired based on sensing by the virtual sensor regarding the virtual object, and The above-mentioned controlling step is a method for controlling a moving body in the space to avoid the virtual object or interact with the virtual object based on the virtual sensing data.

4. In Paragraph 2, A step of acquiring actual sensing data by sensing the space by at least one sensor of the moving body A method for controlling a moving object, further comprising 5. In Paragraph 4, The above sensor and the above virtual sensor are sensors of different types, or, A method for controlling a moving object, wherein the above sensor and the above virtual sensor are sensors of the same different type but have different detection ranges.

6. In Paragraph 4, A step comprising controlling the above operation, wherein, according to a predetermined condition, the operation of the moving body within the space is controlled using at least one of the actual sensing data or the virtual sensing data. A method for controlling a moving object, including 7. In Paragraph 6, According to the above predetermined conditions, the step of controlling the operation of the moving body within the space using at least one of the actual sensing data and the virtual sensing data is If the above actual sensing data and the above virtual sensing data are of the same type, A method for controlling a moving body in a space, based on having a larger or smaller value between the actual sensing data and the virtual sensing data.

8. In Paragraph 6, The above actual sensing data is data sensed by a camera as the actual sensor, and the above virtual sensing data is data sensed by a virtual distance sensor. According to the above predetermined conditions, the step of controlling the operation of the moving body within the space using at least one of the actual sensing data and the virtual sensing data is A method for controlling a moving body, which controls an operation including driving within the space using both the actual sensing data and the virtual sensing data.

9. In Paragraph 6, According to the above predetermined conditions, the step of controlling the operation of the moving body within the space using at least one of the actual sensing data and the virtual sensing data is A method for controlling a moving body, wherein when the moving body is at a specific location within the space or enters a specific area, the movement of the moving body within the space is controlled using virtual sensing data instead of actual sensing data.

10. In Paragraph 6, According to the above predetermined conditions, the step of controlling the operation of the moving body within the space using at least one of the actual sensing data and the virtual sensing data is A step of masking data among the actual sensing data and the virtual sensing data that is not used to control the operation of the mobile body within the space, or overriding the unused data with data among the actual sensing data and the virtual sensing data that is used to control the operation of the mobile body within the space. A method for controlling a moving object, including 11. In Paragraph 3, The above-mentioned controlling step is, A step of providing information about the virtual object or the virtual environment through the moving body based on the virtual sensing data. A method for controlling a moving object, including 12. In Paragraph 11, A method for controlling a moving object, wherein the virtual object above corresponds to an AR (Augmented Reality) object displayed in an AR view corresponding to the virtual environment.

13. In Paragraph 3, A method for controlling a moving object, wherein the virtual object includes a virtual wall that blocks the movement of the moving object.

14. In Paragraph 3, The above virtual sensing data includes data representing a repulsive force toward the virtual object, and The above-mentioned controlling step is a method for controlling a moving body by controlling the moving body with a force in a direction away from the virtual object based on the virtual sensing data.

15. In Paragraph 1, The virtual sensing data above includes data representing a virtual resistance for decelerating the moving body within the space, and The above-mentioned controlling step is a method for controlling a moving body so that the moving body decelerates based on the above-mentioned virtual sensing data.

16. In Paragraph 1, The above-mentioned acquisition step involves acquiring updated virtual sensing data for the updated virtual environment as the virtual environment is updated, and The above-mentioned controlling step is a method for controlling a moving body, wherein the moving body controls the movement of the moving body within the space based on the above-mentioned updated virtual sensing data.

17. A program stored on a computer-readable recording medium that executes the method for controlling a moving body according to claim 1 on a computer system constituting the moving body or the control system.

18. In a computer system constituting a moving body or a control system controlling said moving body, At least one processor implemented to execute computer-readable instructions Includes, The above at least one processor is, Control the moving object within the space, A computer system that acquires virtual sensing data for a simulated virtual environment corresponding to the space and controls the operation of the moving body within the space based on the virtual sensing data.

Images