Network management device, network management method, and network management program
Patent Information
- Application Number
- PCT/JP2025/019401
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-02-26
- Filing Date
- 2025-05-29
- Publication Date
- 2026-09-03
Smart Images
Figure JP2025019401_03092026_PF_FP_ABST
Abstract
Description
Network management apparatus, network management method and network management program
[0001] The present disclosure relates to a time synchronization technique in a virtualized control system such as FA. FA is an abbreviation for Factory Automation.
[0002] In the FA field, transition to a control system configuration in which various functions are installed on the same general-purpose server using container virtualization technology or the like has begun. The various functions are, for example, functions of a controller such as a PLC used for facility control in a production line, and functions of related information system applications. PLC is an abbreviation for Programmable Logic Controller.
[0003] A container environment is constructed using a plurality of servers, and scalable software services may be provided using container orchestration technology such as Kubernetes. In this software service, in consideration of performance, cost, and the like, a base where an application is placed and an amount of resources to be allocated to the application are determined according to a user's request (see Patent Document 1).
[0004] TSN exists as one of industrial networks. TSN is an abbreviation for Time-Sensitive Networking. TSN is a general term for various standards for guaranteeing the real-time performance of control communication in order to realize control of in-factory control devices that require real-time performance on Ethernet. Guaranteeing real-time performance means guaranteeing low-latency and low-jitter communication. The following configuration is a typical implementation example of TSN. High-precision time synchronization on the order of nanoseconds is realized between nodes using IEEE 1588, IEEE 802.1AS, or the like. Then, the transmission timing of an Ethernet frame is realized by a combination with IEEE 802.1Qbv.
[0005] In network time synchronization functions using IEEE 1588 or IEEE 802.1AS, the following processes (1) to (3) are performed. This makes it possible for all nodes on the same clock domain to use common time information. (1) A time synchronization master server is selected from among the clock domains to which it belongs. The time synchronization master server is also called the grandmaster. (2) The time synchronization master server distributes time information to the group of nodes that belong to the same clock domain. (3) Time synchronization slave servers, which are other nodes belonging to the same clock domain, correct their system clocks to match the time of the time synchronization master server.
[0006] Japanese Patent Publication No. 2024-141251
[0007] The technology described in Patent Document 1 may result in time inconsistencies. Specifically, containerization may create a configuration where containers belonging to different clock domains are mixed on the same server. In such a configuration, each time synchronization slave container corrects its system clock to match the time of the time synchronization master container of its respective clock domain. Here, the system clock is unique to each server. Therefore, time synchronization slave containers belonging to different clock domains located on the same server will be correcting the same system clock. As a result, the corrections made to the server's system clock may conflict, potentially leading to time inconsistencies.
[0008] This disclosure aims to prevent time inconsistencies in environments where containers belonging to different clock domains are mixed on a single server.
[0009] The network management device relating to this disclosure includes a master container selection unit that selects one of the containers belonging to a selected clock domain as a time synchronization master container, with each of the multiple clock domains to which multiple containers operating on multiple information processing devices constituting a computer system belong as the selection target, and a master server selection unit that selects an information processing device from among the multiple information processing devices whose clock specifications meet the criteria as a time synchronization master server on which the time synchronization master containers selected by the master selection unit for each of the multiple clock domains are located.
[0010] In this disclosure, a time synchronization master container is selected for each clock domain, and the time synchronization master containers for each clock domain are placed on a time synchronization master server. This ensures that the same time information is used throughout the computer system, thereby avoiding time inconsistencies.
[0011] Configuration diagram of the computer system 10 according to Embodiment 1. Configuration diagram of the information processing device 100 according to Embodiment 1. Configuration diagram of the time synchronization function management unit 200 according to Embodiment 1. Configuration diagram of the device sharing management unit 300 according to Embodiment 1. Configuration diagram of the TSN data transmission management unit 400 according to Embodiment 1. Configuration diagram of the resource allocation management unit 500 according to Embodiment 1. Explanatory diagram of the migration exclusion list 240 according to Embodiment 1. Explanatory diagram of the bandwidth utilization list 350 according to Embodiment 1. Explanatory diagram of the shareable container list 360 according to Embodiment 1. Explanatory diagram of the resource utilization list 540 according to Embodiment 1. Explanatory diagram of the collected data 550 according to Embodiment 1. Flowchart of the trial operation phase according to Embodiment 1. Flowchart of the time synchronization master container allocation phase according to Embodiment 1. Flowchart of the container and resource allocation phase according to Embodiment 1. Flowchart of the actual operation phase according to Embodiment 1. Flowchart of the time synchronization master container allocation phase according to Embodiment 2. Flowchart of the time synchronization master container allocation phase according to Embodiment 3. Flowchart of the time synchronization master container allocation phase according to Embodiment 4.
[0012] Embodiment 1. ***Description of Configuration*** Referring to Figure 1, the configuration of the computer system 10 according to Embodiment 1 will be described. The computer system 10 comprises a plurality of information processing devices 100 and a network management device 1000. The information processing device 100 is a computer on which a real-time container 103 on which a real-time application 101 runs and a general-purpose container 104 on which a general-purpose application 102 runs. The network management device 1000 is a computer that manages network devices. Note that the network management device 1000 may be implemented as one of the information processing devices 100.
[0013] Referring to Figure 2, the configuration of the information processing device 100 according to Embodiment 1 will be described. The information processing device 100 is a computer that operates one or more real-time containers 103 and one or more general-purpose containers 104. As shown in Figure 2, the information processing device 100 operates software such as the real-time container 103, the general-purpose container 104, and virtualization software 105. The real-time container 103 is a container that executes a real-time application 101 that requests TSN communication. The general-purpose container 104 is a container that executes a general-purpose application 102 such as a non-real-time information system application.
[0014] The information processing device 100 includes multiple types of hardware, such as a processor 106, memory 107, I / O device 108, graphics device 109, and network device 110. I / O stands for Input / Output.
[0015] The real-time container 103 executes the real-time application 101 and uses the network device 110 to perform control processing using TSN with connected I / O devices 108, etc. The general-purpose container 104 executes the general-purpose application 102 and performs non-real-time calculation processing, etc. The virtualization software 105 is software equivalent to the container runtime.
[0016] The processor 106 is connected to other hardware such as the memory 107, I / O device 108, graphics device 109, and network device 110 via the bus 120, and controls this other hardware. The hardware connected via the bus 120 can send and receive data to and from each other under the management of the processor 106.
[0017] The processor 106 is, for example, a multi-core CPU, and executes software such as a real-time container 103, a general-purpose container 104, and virtualization software. CPU stands for Central Processing Unit. Memory 107 is, in specific cases, RAM. I / O devices 108 are, in specific cases, a keyboard and a display. Graphics device 109 is a so-called video card that has functions for displaying graphics implemented. Network device 110 is, in specific cases, a network interface card, etc.
[0018] The information processing device 100 is provided with a transmission management buffer 111, which is an area of memory 107 for holding transmission data. The transmission management buffer 111 is provided in a quantity equal to the number of network devices 110 mounted on the information processing device 100.
[0019] Communication with containers or external I / O devices 108 mounted on other information processing devices 100 is achieved via TSN communication through the network device 110.
[0020] Referring to Figures 3 to 6, the functional components of the network management device 1000 will be described. As shown in Figure 1, the network management device 1000 includes a time synchronization function management unit 200, a device sharing management unit 300, a TSN data transmission management unit 400, and a resource allocation management unit 500 as functional components. Each functional component of the network management device 1000 is implemented by software.
[0021] Referring to Figure 3, the time synchronization function management unit 200 according to Embodiment 1 will be described. The time synchronization function management unit 200 is a function that fixes the time synchronization master container and excludes the time synchronization master container from migration, thereby preventing the switching of the time synchronization master server. The time synchronization function management unit 200 includes a master container selection unit 210, a master server selection unit 220, and a master setting management unit 230 as functional components. The time synchronization function management unit 200 stores a migration exclusion list 240 in a storage device.
[0022] Referring to Figure 4, the device sharing management unit 300 according to Embodiment 1 will be described. The device sharing management unit 300 has a function to calculate a communication cycle indicating a recommended time slot if it determines that it is difficult to complete communication within a specified time slot, and to present this to the user. The device sharing management unit 300 also has a function to identify combinations that can share a single network device 110 and to determine whether data can be transmitted without loss within a specified time slot. The device sharing management unit 300 includes, as functional components, a bandwidth usage data acquisition unit 310, a sharing configuration identification unit 320, a data loss determination unit 330, and a communication cycle change unit 340. The device sharing management unit 300 stores a bandwidth usage rate list 350 and a shareable container list 360 in a storage device.
[0023] Referring to Figure 5, the TSN data transmission management unit 400 according to Embodiment 1 will be described. The TSN data transmission management unit 400 has the function of centrally managing the transmission data of containers that share a single network device 110 and centrally accessing the network device 110. The TSN data transmission management unit 400 includes a data transmission detection unit 410, a transmission data storage unit 420, and a transmission execution unit 430 as functional components.
[0024] Referring to Figure 6, the resource allocation management unit 500 according to Embodiment 1 will be described. The resource allocation management unit 500 has the function of managing the resources to be allocated to each container. The resource allocation management unit 500 includes, as functional components, a resource utilization rate collection unit 510, an evaluation parameter identification unit 520, and an allocation identification unit 530. The resource allocation management unit 500 stores a resource utilization rate list 540 and collected data 550 in a storage device.
[0025] Referring to Figure 7, the migration exclusion list 240 according to Embodiment 1 will be explained. The migration exclusion list 240 registers information for identifying the time synchronization master container in order to set the time synchronization master container as excluded from migration. In Figure 7, the container ID of the time synchronization master container is registered as the information for identifying the container.
[0026] Referring to Figure 8, the bandwidth utilization list 350 according to Embodiment 1 will be described. The bandwidth utilization list 350 registers the bandwidth utilization of each network device 110 for each container, which has been collected by the bandwidth utilization data acquisition unit 310. The bandwidth utilization list 350 also registers the date and time the bandwidth utilization was acquired as a timestamp. In Figure 8, one or more pairs of the container ID of the time-synchronized master container and the bandwidth utilization are registered for each timestamp.
[0027] Referring to Figure 9, the shareable container list 360 according to Embodiment 1 will be described. The shareable container list 360 registers information for identifying shareable containers identified by the shared configuration identification unit 320 for each network device 110. In Figure 9, a container ID is registered for each device name of the information processing device 100 and the network device 110.
[0028] Referring to Figure 10, the resource usage list 540 according to Embodiment 1 will be described. The resource usage list 540 registers the hardware resource usage collected by the resource usage collection unit 510 for each container. In addition, the resource usage list 540 registers the date and time the resource usage was acquired as a timestamp along with the resource usage. In Figure 10, for each timestamp, the container ID, the server ID which is the identification information of the information processing device 100 on which the container is located, and a set of resource usage consisting of the CPU usage rate 106, the memory usage rate 107, and the power consumption are registered.
[0029] Referring to Figure 11, the collected data 550 according to Embodiment 1 will be described. In the collected data 550, a data type is registered for each user request. User requests are priorities regarding container placement or device assignment defined by the user from an engineering tool or the like. Possible user requests include arranging so that the CPU usage limit is 70%, and prioritizing placing containers on the same information processing device 100 as much as possible. The data type is the resource usage of the hardware related to the user request.
[0030] ***Explanation of Operation*** The operation of the network management device 1000 according to Embodiment 1 will be explained with reference to Figures 12 to 15. The operation procedure of the network management device 1000 according to Embodiment 1 corresponds to the network management method according to Embodiment 1. Furthermore, the program that realizes the operation of the network management device 1000 according to Embodiment 1 corresponds to the network management program according to Embodiment 1.
[0031] The operation of the network management device 1000 according to Embodiment 1 includes four phases: a trial operation phase, a time synchronization master container placement phase, a container and resource placement phase, and an actual operation phase.
[0032] Referring to Figure 12, the trial operation phase according to Embodiment 1 will be described. (Step S101: Parameter Identification Processing) The evaluation parameter identification unit 520 receives a user request. The evaluation parameter identification unit 520 identifies the parameters to be collected in accordance with the user request. For each user request, the evaluation parameter identification unit 520 registers the identified parameters in the collected data 550. User requests may be input through an engineering tool or the like, or they may be stored in a parameter file or the like.
[0033] (Step S102: Trial run process) A trial run of the computer system 10 is performed. Specifically, all real-time applications 101 and general-purpose applications 102 that are executed in the actual operation phase are executed. This realizes an environment identical to that of actual operation. The applications that are executed include, for example, an application to copy data to memory 107, an application to acquire sensor information, and an application to analyze data. These applications are executed with the aim of acquiring information on the hardware resource usage status during actual system operation.
[0034] Furthermore, if the resource usage rates indicated by each data type registered in the collected data 550 can be collected, then only some applications may be executed instead of all applications.
[0035] (Step S103: Data collection process) The resource utilization rate collection unit 510 collects the resource utilization rates indicated by each data type registered in the collected data 550 during the trial run performed in step S102.
[0036] (Step S104: Data registration process) The resource usage rate collection unit 510 registers the resource usage rate indicated by each data type collected in step S103 into the resource usage rate list 540.
[0037] Referring to Figure 13, the time synchronization master container placement phase according to Embodiment 1 will be described. (Step S201: Setting acquisition process) The master container selection unit 210 acquires a configured network configuration file from the engineering tool. The network configuration file contains settings related to the TSN network.
[0038] (Step S202: Master Container Selection Process) The master container selection unit 210 sets each of the multiple clock domains to which the multiple containers operating on the multiple information processing devices 100 constituting the computer system 10 belong as selection targets. At this time, the master container selection unit 210 identifies each clock domain in the computer system 10 by referring to the network configuration file obtained in step S201. The master container selection unit 210 detects the containers present in the clock domains to be selected by referring to the network configuration file obtained in step S201. Then, the master container selection unit 210 selects one of the detected containers as the time synchronization master container.
[0039] (Step S203: Master Server Selection Process) The master server selection unit 220 acquires specification information indicating the specifications of the clocks installed in each of the multiple information processing devices 100 provided by the computer system 10. The master server selection unit 220 selects the information processing device 100 from among the multiple information processing devices 100 whose specifications indicated by the clock specification information meet the criteria as the time synchronization master server. Here, the master server selection unit 220 selects the information processing device 100 with the highest accuracy of the installed clock from among the multiple information processing devices 100 as the time synchronization master server. The time synchronization master server is the information processing device 100 on which the time synchronization master containers selected for each of the multiple clock domains in step S202 are located.
[0040] (Step S204: Placement Determination Process) The master setting management unit 230 determines whether all of the time synchronization master containers selected for each of the multiple clock domains in step S202 can be placed on the time synchronization master server selected in step S203. At this time, the master setting management unit 230 refers to the resource utilization list 540 registered in step S104 and determines whether all of the time synchronization master containers for all clock domains can be placed on the time synchronization master server. If placement is possible, the master setting management unit 230 proceeds to step S205. On the other hand, if placement is not possible, the master setting management unit 230 proceeds to step S208.
[0041] This section describes a method for determining whether all time synchronization master containers selected for each of the multiple clock domains can be placed on the time synchronization master server. The master setting management unit 230 determines whether placement is possible by referring to the resource utilization rates of hardware such as the processor 106 and memory 107 listed in the resource utilization list 540. For example, suppose container A, which is allocated to one core on the information processing device 100A, has a processor 106A utilization rate of 30%, and container B, which is allocated to one core on the information processing device 100B, has a processor 106B utilization rate of 40%. The processor 106A utilization rate of container A (30%) + the processor 106B utilization rate of container B (40%) < 100%. Therefore, the master setting management unit 230 determines that the processor 106 can be placed on one core of the time synchronization master server. If the performance of the processor 106 differs in each information processing device 100, the master setting management unit 230 uses the frequency difference to derive an approximate value and, based on the calculated approximate value, determines whether or not it can be placed on the core of the time synchronization master server.
[0042] The master setting management unit 230 similarly determines whether other hardware resources such as the memory 107 can be arranged. For example, assume that it is determined that the time synchronization master containers A, B, and C for respective clock domains can be arranged on the time synchronization master server from the perspective of the usage rate of the processor 106. In this case, if the sum of the memory sizes allocated to each of the time synchronization master containers A, B, and C for the respective clock domains is smaller than the memory size installed in the time synchronization master server, arrangement is also possible from the perspective of the memory 107. That is, if (sum of memory sizes of time synchronization master containers = container A memory size + container B memory size + container C memory size) < memory size installed in the time synchronization master server, arrangement is also possible from the perspective of the memory 107.
[0043] Assume that the total capacity of the memory 107 installed in the time synchronization master server is smaller than the sum of the memory sizes of the time synchronization master containers. In this case, the master setting management unit 230 may change the memory size allocated to each container to determine whether the containers can be installed. For example, although 1 GB of memory is allocated to container A, if the memory usage rate is 10%, it is conceivable to change the allocated memory size from 1 GB to 500 MB. The size to be allocated after the change may be predefined by the user, or may be set by detecting the minimum size for operation through a test run.
[0044] Even when using hardware resources other than the processor 106 and the memory 107, the determination can be made based on the above resource allocation information and the usage rate information of each container. The master setting management unit 230 determines whether arrangement on the time synchronization master server is possible by applying an AND condition to the determination results for each hardware resource.
[0045] (Step S205: Message adjustment process) The master setting management unit 230 modifies the information in the announcement message used for selecting the time synchronization master in order to prevent the time synchronization master server from being changed by the BMCA function. BMCA stands for Best Master Clock Algorithm. The time synchronization master server is changed by the BMCA function used to select the time synchronization master server to be used in the TSN time synchronization function.
[0046] (Step S206: Master Container Placement Process) The master setting management unit 230 instructs the orchestrator to place the time synchronization master containers selected for each of the multiple clock domains in step S202 on the time synchronization master server selected in step S203. The orchestrator migrates and places each time synchronization master container on the time synchronization master server in accordance with the instructions.
[0047] (Step S207: Migration exclusion setting process) The master setting management unit 230 controls the time synchronization master containers selected for each of the multiple clock domains in step S202 so as not to be migrated from the time master server to other information processing devices 100. Specifically, the master setting management unit 230 registers the container IDs of the time synchronization master containers selected for each of the multiple clock domains in step S202 in the migration exclusion list 240. Here, containers whose container IDs are registered in the migration exclusion list 240 are controlled so as not to be migrated.
[0048] (Step S208: Master server addition processing) The master server selection unit 220 additionally selects, from among the plurality of information processing apparatuses 100 other than the time synchronization master server, an information processing apparatus 100 whose specification indicated by the clock specification information satisfies a criterion as a new time synchronization master server. Here, the master server selection unit 220 selects, among the information processing apparatuses 100 other than the time synchronization master server, the information processing apparatus 100 equipped with the highest-accuracy clock as the new time synchronization master server.
[0049] (Step S209: Slave container generation processing) A master configuration management unit 230 generates a time synchronization slave container that performs time synchronization between a plurality of time synchronization master servers. Let the time synchronization master server selected in step S203 be a first time synchronization master server, and the time synchronization master server additionally selected in step S208 be a second time synchronization master server. The time synchronization slave container performs processing for synchronizing the time of the second time synchronization master server to the time of the first time synchronization master server.
[0050] (Step S210: Placement Determination Process) The master setting management unit 230 determines whether the time synchronization master containers selected for each of the multiple clock domains in step S202 and the time synchronization slave containers generated in step S209 can be placed on the time synchronization master server selected in steps S203 and S208. At this time, the master setting management unit 230 refers to the resource usage list 540 registered in step S104 and determines whether the time synchronization master containers and time synchronization slave containers for all clock domains can be placed on the time synchronization master server. Specifically, as described in step S204, the master setting management unit 230 searches for a combination that can place all time synchronization master containers based on the hardware resource usage. If the master setting management unit 230 finds a combination that can place all time synchronization master containers, it determines that placement is possible; otherwise, it determines that placement is not possible. If placement is possible, the master setting management unit 230 proceeds to step S211. On the other hand, if placement is not possible, the master setting management unit 230 returns the process to step S208.
[0051] In the process from step S208 to step S210, a new time synchronization master server is additionally selected from the information processing devices 100 other than the already selected time synchronization master servers until the master setting management unit 230 determines that it can be deployed. In addition, a time synchronization slave container corresponding to the new time synchronization master server is generated.
[0052] (Step S211: Message adjustment process) Similar to step S205, the master setting management unit 230 modifies the information in the announcement message used for selecting the time synchronization master in order to prevent the time synchronization master server from being changed by the BMCA function.
[0053] (Step S212: Master Container Placement Processing) The master configuration management unit 230 instructs the orchestrator to place the time synchronization master containers selected for each of the multiple clock domains on the time synchronization master servers selected in steps S203 and S208. Specifically, the master configuration management unit 230 instructs the time synchronization master server selected in step S203 to place as many time synchronization master containers as possible. The master configuration management unit 230 also instructs the time synchronization master server selected in step S208 to place the time synchronization master containers that could not be placed on the time synchronization master server selected in step S203. Furthermore, the master configuration management unit 230 instructs the time synchronization master server selected in step S208 to place the time synchronization slave containers generated in step S209. The orchestrator migrates and places each time synchronization master container and time synchronization slave container on the time synchronization master server according to the instructions. The orchestrator also allocates resources to each container according to the instructions.
[0054] (Step S213: Processing to exclude from migration) Similar to step S207, the master setting management unit 230 controls the time synchronization master containers selected for each of the multiple clock domains in step S202 so as not to be migrated from the time master server to other information processing devices. Specifically, the master setting management unit 230 registers the container IDs of the time synchronization master containers selected for each of the multiple clock domains in step S202 in the migration exclusion list 240.
[0055] Referring to Figure 14, the container and resource placement phase according to Embodiment 1 will be described. (Step S301: Bandwidth Usage Data Acquisition Process) The Bandwidth Usage Data Acquisition Unit 310 acquires bandwidth usage information for each clock domain, indicating the bandwidth usage rate of the network used by each container in each time slot. The Bandwidth Usage Data Acquisition Unit 310 registers the bandwidth usage rate indicated by the bandwidth usage information in the Bandwidth Usage Rate List 350.
[0056] (Step S302: Sharing permission determination process) The sharing configuration identification unit 320 determines whether or not the network device 110 is permitted to be shared by multiple containers. The setting for permitting or denying the sharing of the network device 110 may be done by an engineering tool or by a parameter file, etc. If permission is granted, the sharing configuration identification unit 320 proceeds to step S303. On the other hand, if permission is denied, the sharing configuration identification unit 320 proceeds to step S306.
[0057] (Step S303: Combination Candidate Generation Process) The shared configuration identification unit 320 identifies candidate combinations of containers that can be shared, based on the network bandwidth usage information of each of the multiple containers, for the network device 110 provided by each information processing device 100. In other words, the shared configuration identification unit 320 identifies candidate combinations of containers that can share a single network device 110, based on the bandwidth usage rates registered in the bandwidth usage rate list 350. Specifically, the shared configuration identification unit 320 identifies combinations of containers in which the sum of the bandwidth usage rates does not exceed 100%. For example, suppose container A has a bandwidth usage rate of 50%, container B has a bandwidth usage rate of 60%, and container C has a bandwidth usage rate of 30%. In this case, the combination of container A and container B would result in a sum of bandwidth usage rates exceeding 100%. Therefore, it is determined that container A and container B cannot be combined. On the other hand, the combination of container A and container C, and the combination of container B and container C, have a sum of bandwidth usage rates of 100% or less. Therefore, the combination of container A and container C, and the combination of container B and container C are determined to be able to share the same network device and are identified as combination candidates. The shared configuration identification unit 320 registers the combination candidates in the shareable container list 360. There is no limit to the number of containers that can share the network device 110.
[0058] (Step S304: Candidate Determination Process) The shared configuration identification unit 320 determines whether or not there is one or more combination candidates in the shareable container list 360. If a combination candidate exists, the shared configuration identification unit 320 proceeds to step S305. On the other hand, if no combination candidate exists, the shared configuration identification unit 320 proceeds to step S306.
[0059] (Step S305: Deployment Plan Identification Process) The shared configuration identification unit 320 identifies candidate information processing devices 100 for each of the multiple containers according to the combination identified in step S303. Specifically, the shared configuration identification unit 320 identifies a deployment plan for the information processing devices 100 for containers that can be allocated other resources such as the processor 106 and memory 107, based on the resource usage rate list 540, for the combination candidates registered in the shareable container list 360. At this time, the shared configuration identification unit 320 checks the resource usage rate of the general-purpose container 104. If the resource usage rate is high, it may cause communication delays in the real-time container 103, so the shared configuration identification unit 320 notifies the user to review the application processing on the general-purpose container 104.
[0060] (Step S306: Exclusive Setting Process) The shared configuration identification unit 320 determines that the network device 110 cannot be shared and sets each container to have exclusive use of the network device 110. After setting, the shared configuration identification unit 320 identifies candidate information processing devices 100 to which containers that can be allocated other resources such as the processor 106 and memory 107 can be placed, based on the resource usage list 540. After that, the shared configuration identification unit 320 proceeds to step S310.
[0061] (Step S307: Determination of the number of layout options) The shared configuration identification unit 320 determines whether or not there are multiple layout options identified in step S305. If there are not multiple layout options, the shared configuration identification unit 320 proceeds to step S308. On the other hand, if there are multiple layout options, the shared configuration identification unit 320 proceeds to step S309.
[0062] (Step S308: Determination of the existence of a configuration plan) The shared configuration identification unit 320 determines whether there is one configuration plan identified in step S305 or whether there are none. If there is one configuration plan, the shared configuration identification unit 320 proceeds to step S310. On the other hand, if there are no configuration plans, the shared configuration identification unit 320 determines that sharing of the network device 110 is not possible and proceeds to step S306.
[0063] (Step S309: Placement Determination Process) The shared configuration identification unit 320 determines which placement plan to adopt from among multiple placement plans based on the user request registered in the collected data 550. Then, the placement identification unit 530 refers to the resource utilization list 540 and determines the allocation of hardware resources such as the processor 106 and memory 107 to each container. Specifically, the placement identification unit 530 determines the placement plan so as to satisfy the user request. For example, suppose the user request is that the utilization rate of the processor 106 is 80% or less. Also, suppose that the utilization rate of the processor 106 in container A is a maximum of 40%, the utilization rate of the processor 106 in container B is a maximum of 50%, and the utilization rate of the processor 106 in container C is a maximum of 30%. In this case, the sum of the utilization rates of the processor 106 for the combination of container A and container B is 90%, the sum of the utilization rates of the processor 106 for the combination of container A and container C is 70%, and the sum of the utilization rates of the processor 106 for the combination of container B and container C is 80%. In this case, the placement determination unit 530 preferentially adopts either the combination of container A and container C, or the combination of container B and container C, as the placement plan. For example, the placement determination unit 530 decides on the placement plan that adopts the combination of container A and container C, which has a smaller total utilization rate of processor 106. Then, regarding the combination of container A and container C, the placement determination unit 530 allocates 40% to container A and 30% to container C.
[0064] (Step S310: Data Loss Determination Process) The data loss determination unit 330 predetermines whether or not data loss will occur in the transmitted data when multiple containers are placed on the destination information processing device 100 according to the arrangement plan determined in step S309. Whether or not data loss will occur in the transmitted data can be determined, for example, by assigning a serial number to the data and performing the transmission process. In other words, by checking the data frames transmitted from the network device 110 using the packet capture function, it can be determined that data loss has occurred if there is a missing serial number, and that no data loss has occurred if there is no missing serial number. The data loss determination unit 330 temporarily applies the arrangement plan and runs the computer system 10 to determine whether or not data loss will occur. In other words, the data loss determination unit 330 temporarily places the containers according to the arrangement plan and allocates hardware resources, and then determines whether or not data loss will occur. If no data loss occurs in the transmitted data, the data loss determination unit 330 proceeds to step S311. On the other hand, if data loss occurs in the transmitted data, the data loss determination unit 330 proceeds to step S312.
[0065] (Step S311: Placement Determination Process) The shared configuration identification unit 320 instructs the orchestrator to place multiple containers on the information processing device 100 at the destination and to allocate hardware resources, according to the placement plan determined in step S309. The orchestrator places the containers on the information processing device 100 and allocates resources in accordance with the instructions.
[0066] (Step S312: Deletion of layout plan) The shared configuration identification unit 320 deletes the layout plan determined in step S309.
[0067] (Step S313: Determination of remaining layout options) The shared configuration identification unit 320 determines whether or not there are any layout options remaining. If there are any layout options remaining, the shared configuration identification unit 320 returns to step S309 to decide which layout option to adopt from the remaining options and to decide on resource allocation. On the other hand, if there are no layout options remaining, the shared configuration identification unit 320 proceeds to step S314.
[0068] (Step S314: Period Change Determination Process) The communication period change unit 340 determines whether it is possible to change the communication period by at least some of the multiple containers. If it is possible to change the communication period, the communication period change unit 340 proceeds to step S315. On the other hand, if it is not possible to change the communication period, the communication period change unit 340 proceeds to step S306 because sharing of the network device 110 is not possible.
[0069] (Step S315: Period Change Determination Process) The communication period change unit 340 changes the communication period for at least some of the multiple containers by redesigning the communication period of the entire computer system 10 by referring to the bandwidth usage rate in the bandwidth usage rate list 350. Specifically, the communication period change unit 340 calculates the amount of data to be transferred for each container that wants to share from the bandwidth and bandwidth usage rate. The communication period change unit 340 calculates the transmission time required for communication for containers that want to share one network device 110 based on the total amount of data and bandwidth information. Here, the communication period change unit 340 changes the calculated transmission time to a time that can be realized by multiplying it by a set value (for example, 1.1 times).
[0070] (Step S316: Data Loss Determination Process) The data loss determination unit 330 determines whether or not data loss occurs in the transmitted data at the communication cycle changed in step S315. If no data loss occurs in the transmitted data, the data loss determination unit 330 proceeds to step S317. On the other hand, if data loss occurs in the transmitted data, the data loss determination unit 330 determines that sharing of the network device 110 is not possible, and therefore proceeds to step S306.
[0071] (Step S317: Change proposal processing) The communication cycle change unit 340 proposes a change to the user as a proposed change to the communication cycle changed in step S315. If the proposed change is accepted, the orchestrator is instructed to adopt the proposed change and, in accordance with the arrangement plan determined in step S309, to arrange multiple containers on the destination information processing device 100 and to allocate hardware resources.
[0072] Referring to Figure 15, the actual operation phase according to Embodiment 1 will be described. (Step S401: Buffer generation process) The transmission data storage unit 420 generates a transmission management buffer 111 for each network device 110 that can store data for each type of data to be used.
[0073] (Step S402: Transmission Request Acquisition Process) The data transmission detection unit 410 traps the transmission requests of each container and notifies the transmission data storage unit 420.
[0074] (Step S403: Type determination process) The transmission data storage unit 420 determines the type of transmission data requested in the transmission request notified in step S402.
[0075] (Step S404: Data storage process) The transmission data storage unit 420 stores the transmission data requested in the transmission request notified in step S402 in the transmission management buffer 111 corresponding to the type determined in step S403.
[0076] (Step S405: Transmission execution process) The transmission execution unit 430 writes the transmission data stored in the transmission management buffer 111 to the hardware transmission queue of the network device 110. Here, the transmission execution unit 430 writes the transmission data stored in the transmission management buffer 111 to the hardware transmission queue of the network device 110 at regular intervals. This enables periodic data transmission.
[0077] ***Effects of Embodiment 1*** As described above, the network management device 1000 according to Embodiment 1 selects a time synchronization master container for each clock domain and places the time synchronization master containers of each clock domain on the time synchronization master server. This makes it possible to use the same time information in the computer system and to avoid time inconsistencies.
[0078] Furthermore, in the case where all time synchronization master containers cannot be placed on a single time synchronization master server, the network management device 1000 according to Embodiment 1 selects multiple time synchronization master servers and performs time synchronization using multiple time synchronization master servers. This ensures that the same time information is used throughout the entire computer system 10, even when all time synchronization master containers cannot be placed on a single time synchronization master server, thereby avoiding time inconsistencies.
[0079] Furthermore, the network management device 1000 according to Embodiment 1 allows the network device 110 to be shared by multiple containers, thereby easing the constraint on the number of containers that can be placed on a single information processing device 100 and enabling effective utilization of the network device 110.
[0080] Embodiment 2. Embodiment 2 describes the process that occurs when an abnormality occurs in the time synchronization master container after unifying the time information of all clock domains using the method described in Embodiment 1. In Embodiment 2, time inconsistencies are avoided by re-selecting the time synchronization master container and migrating to the time synchronization master server. Embodiment 2 describes the differences from Embodiment 1, and omits the explanation of the same points.
[0081] ***Explanation of Operation*** The time synchronization master container placement phase according to Embodiment 2 will be explained with reference to Figure 16. The process in step S501 is the same as step S201 in Figure 13.
[0082] (Step S502: Container detection process) The master container selection unit 210 sets each of the multiple clock domains as a selection target, similar to step S202 in Figure 13. The master container selection unit 210 refers to the network configuration file obtained in step S501 and detects the containers present in the clock domains to be selected.
[0083] (Step S503: Master container deletion process) The master setting management unit 230 deletes the time synchronization master container in which the abnormality occurred from the time synchronization master server.
[0084] (Step S504: List Deletion Process) The master setting management unit 230 deletes the time synchronization master container in which the abnormality occurred from the list of items excluded from migration 240.
[0085] (Step S505: Container Determination Process) The master container selection unit 210 determines whether a container (time synchronization slave container) belonging to the same clock domain as the time synchronization master container where the abnormality occurred is installed on any of the time synchronization master servers. If a container is installed, the master container selection unit 210 proceeds to step S506. On the other hand, if no container is installed, the master container selection unit 210 proceeds to step S507.
[0086] (Step S506: Container change process) The master container selection unit 210 changes one container that belongs to the same clock domain as the time synchronization master container where the abnormality occurred and is installed on the time synchronization master server to the time synchronization master container. Then, the master container selection unit 210 proceeds to step S205 in Figure 13, and the same process is executed.
[0087] (Step S507: Container Selection Process) The master container selection unit 210 selects one container as the time synchronization master container from among the containers belonging to the same clock domain as the time synchronization master container where the abnormality occurred.
[0088] (Step S508: Placement Determination Process) The master setting management unit 230 determines whether the time synchronization master container selected in step S507 can be placed on any of the time synchronization master servers. If placement is possible, the master setting management unit 230 proceeds to step S205 in Figure 13, and the same process is executed. On the other hand, if placement is not possible, the master setting management unit 230 proceeds to step S208 in Figure 13, and the same process is executed.
[0089] ***Effects of Embodiment 2*** As described above, the network management device 1000 according to Embodiment 2 selects the next time synchronization master container if an abnormality occurs in the time synchronization master container that has been selected once. This makes it possible to continuously use the same time throughout the entire system. As a result, in a configuration where containers belonging to different clock domains are placed on the same server, even if an abnormality occurs in the time synchronization master container, it is possible to prevent time correction conflicts and continue processing of the real-time container 103.
[0090] Embodiment 3. Embodiment 3 describes the process when a new information processing device 100 is added after the time information of all clock domains has been unified to the time of the time synchronization master server using the method described in Embodiment 1. Embodiment 3 describes the differences from Embodiment 1, and omits the explanation of the same points.
[0091] ***Explanation of Operation*** Referring to Figure 17, the time synchronization master container placement phase according to Embodiment 3 will be explained. (Step S601: Detection Process) The master container selection unit 210 detects that an information processing device 100 has been newly added to the computer system 10. This detection method can be any method, and existing technology may be used.
[0092] (Step S602: Specification acquisition process) The master server selection unit 220 acquires specification information indicating the specifications of the clock installed in the added information processing device 100.
[0093] (Step S603: Specification Comparison Process) The master server selection unit 220 determines whether the clock specifications of the added information processing device 100 are better than those of the current time synchronization master server. Specifically, the master server selection unit 220 determines whether the clock accuracy indicated by the specification information obtained in step S602 is higher than the accuracy of the clock installed in the current time synchronization master server. The current time synchronization master server here is the time synchronization master server selected in step S203 in Figure 13. If the clock installed in the added information processing device 100 is better, that is, if the clock accuracy is higher, the master server selection unit 220 proceeds to step S604. On the other hand, if the added information processing device 100 is not better, that is, if the clock accuracy is not higher, the master server selection unit 220 terminates the process because there is no need to change the time synchronization master server.
[0094] (Step S604: First Placement Determination Process) The master setting management unit 230 determines whether the time synchronization master container currently placed on the time synchronization master server can be placed on the added information processing device 100. If it can be placed, the master setting management unit 230 proceeds to step S211 in Figure 13, and the same process is executed. On the other hand, if it cannot be placed, the master setting management unit 230 proceeds to step S605.
[0095] (Step S605: Second Placement Determination Process) The master setting management unit 230 determines whether time synchronization master containers that cannot be placed on the added information processing device 100 can be placed on other time synchronization master servers. If placement is possible, the master setting management unit 230 proceeds to step S211 in Figure 13, and the same process is executed. On the other hand, if placement is not possible, the master setting management unit 230 proceeds to step S208 in Figure 13, and the same process is executed.
[0096] ***Effects of Embodiment 3*** As described above, the network management device 1000 according to Embodiment 3 replaces the time synchronization master server when an information processing device 100 with high clock accuracy is added. This makes it possible to achieve time synchronization using a more accurate clock.
[0097] Embodiment 4. Embodiment 4 describes the process when a new clock domain is added after the time information of all clock domains has been unified to the time of the time synchronization master server using the method described in Embodiment 1. Embodiment 4 describes the differences from Embodiment 1, and omits the explanation of the same points.
[0098] ***Explanation of Operation*** Referring to Figure 18, the time synchronization master container placement phase according to Embodiment 4 will be explained. (Step S701: Detection Process) The master container selection unit 210 detects that a new clock domain has been added to the computer system 10. This detection method can be any method, and existing technology may be used.
[0099] (Step S702: Configuration acquisition process) Similar to step S201 in Figure 13, the master container selection unit 210 acquires the configured network configuration file from the engineering tool.
[0100] (Step S703: Master Container Selection Process) The master container selection unit 210 sets the added clock domain as the selection target. The master container selection unit 210 detects the containers present in the clock domain to be selected, similar to step S202 in Figure 13, and selects one of the containers as the time synchronization master container.
[0101] (Step S704: Placement Determination Process) The master setting management unit 230 determines whether the time synchronization master container selected in step S703 can be placed on any of the time synchronization master servers. If placement is possible, the master setting management unit 230 proceeds to step S211 and the same process is executed. On the other hand, if placement is not possible, the master setting management unit 230 proceeds to step S208 and the same process is executed.
[0102] ***Effects of Embodiment 4*** As described above, the network management device 1000 according to Embodiment 4 selects a time synchronization master container in the newly added clock domain when a new clock domain is added. This makes it possible to continuously use the same time throughout the entire computer system 10. As a result, even in a configuration where containers belonging to different clock domains are placed on the same server, when a clock domain is added, time correction conflicts are prevented and processing of the real-time container 103 can continue.
[0103] ***Other Configurations*** <Variation 1> The real-time container 103 and the general-purpose container 104 may be placed in the same information processing device 100, or multiple containers may be placed in the same information processing device 100. Also, the network management device 1000 may be one of the information processing devices 100.
[0104] <Modification 2> In Embodiment 1, each functional component of the network management device 1000 is implemented in software. In this case, the program that implements each functional component of the network management device 1000 is read and executed by the processor of the network management device 1000, thereby realizing the function of each functional component. As Modification 2, each functional component may be implemented in hardware. In this case, the network management device 1000 includes an electronic circuit that implements the function of each functional component instead of a processor. Examples of electronic circuits include single circuits, composite circuits, programmed processors, parallel programmed processors, logic ICs, GAs, ASICs, and FPGAs. GA stands for Gate Array. ASIC stands for Application Specific Integrated Circuit. FPGA stands for Field-Programmable Gate Array. Each functional component may be implemented in a single electronic circuit, or it may be implemented by distributing each functional component across multiple electronic circuits.
[0105] <Modification 3> As Modification 3, some of the functional components may be implemented in hardware, while other functional components may be implemented in software.
[0106] A processor and electronic circuitry together are called a processing circuit. In other words, the function of each functional component is realized by the processing circuit.
[0107] The various aspects of this disclosure are summarized below as appendices. (Appendix 1) A network management device comprising: a master container selection unit that selects one of the containers belonging to a selected clock domain as a time synchronization master container, with each of the multiple clock domains to which a plurality of containers operating on a plurality of information processing devices constituting a computer system belong as the selection target; and a master server selection unit that selects an information processing device from among the plurality of information processing devices whose clock specifications meet the criteria as a time synchronization master server on which the time synchronization master container selected by the master container selection unit for each of the plurality of clock domains is located. (Appendix 2) The network management device further comprises a master setting management unit that determines whether the time synchronization master container selected for each of the plurality of clock domains can be located on the time synchronization master server selected by the master server selection unit, and the network management device according to Appendix 1, wherein the master server selection unit additionally selects a new time synchronization master server on which the time synchronization master container is located from among the plurality of information processing devices other than the time synchronization master server until it is determined by the master setting management unit that it can be located. (Note 3) The network management device described in Note 2, wherein the master server selection unit generates a time synchronization slave container that performs time synchronization between a plurality of time synchronization master servers. (Note 4) The network management device described in Note 2 or 3, wherein the master setting management unit determines whether or not the time synchronization master container can be placed on the time synchronization master server based on the usage rate of hardware resources when the computer system is running. (Note 5) The network management device described in Note 1, further comprising a master setting management unit that controls the time synchronization master container not to be migrated from the time master server to an information processing device other than the time master server.(Note 6) The network management device further comprises a shared configuration identification unit that identifies a combination of containers that can share a network device provided by any of the multiple information processing devices based on the network bandwidth usage information of each of the multiple containers, and identifies an information processing device to which each of the multiple containers is to be placed according to the identified combination. (Note 7) The network management device according to Note 6, wherein the shared configuration identification unit identifies an information processing device to which each of the multiple containers is to be placed in order to satisfy hardware resource requirements. (Note 8) The network management device according to Note 6 or 7, further comprising a data loss determination unit that determines whether or not data loss occurs in transmitted data when the multiple containers are placed in the information processing device to which they are to be placed identified by the shared configuration identification unit. (Note 9) The network management device according to Note 8, further comprising a communication cycle modification unit that changes the communication cycle of at least some of the multiple containers when the data loss determination unit determines that data loss occurs, and the data loss determination unit determines whether or not data loss occurs in transmitted data when the communication cycle modified by the communication cycle modification unit is applied. (Note 10) The network management device according to any one of Notes 1 to 9, wherein when an abnormality occurs in the time synchronization master container for the clock domain to be processed among the plurality of clock domains, if there is a container located on the time synchronization master server among the containers belonging to the clock domain to be processed, the master container selection unit selects that container as the new time synchronization master container, and if there is no container located on the time synchronization master server among the containers belonging to the clock domain to be processed, it selects any container from among the containers belonging to the clock domain to be processed as the new time synchronization master container and places it on the time synchronization master server.(Note 11) The network management device according to any one of Notes 1 to 10, wherein when a new information processing device is added to the computer system, the master server selection unit selects the new information processing device as the new time synchronization master server in place of the time synchronization master server if the new information processing device has better clock specifications than the time synchronization master server. (Note 12) The network management device according to Note 2, wherein when a new clock domain is added, the master container selection unit selects one of the containers belonging to the new clock domain as the time synchronization master container, and the master setting management unit determines whether the time synchronization master container for the new clock domain can be placed on the time synchronization master server. (Note 13) A network management method wherein a computer selects each of several clock domains to which several containers operating on several information processing devices constituting a computer system belong as a selection target, and selects one of the containers belonging to the selected clock domain as a time synchronization master container, and the computer selects an information processing device from among the several information processing devices whose clock specifications meet the criteria as a time synchronization master server to which the time synchronization master containers selected for each of the several clock domains are located. (Note 14) A network management program that causes a computer to function as a network management device that performs a master container selection process in which a master container selection process is performed in which a master container selection process is performed in which a master container selection process is performed in which a master server selection process is performed in which a master server selection process is performed in which a master processing device from among the several information processing devices whose clock specifications meet the criteria is selected as a time synchronization master server to which the time synchronization master containers selected for each of the several clock domains are located.
[0108] The embodiments and variations of this disclosure have been described above. Some of these embodiments and variations may be implemented in combination. Alternatively, some or all of them may be implemented in part. However, this disclosure is not limited to the embodiments and variations described above, and various modifications are possible as needed.
[0109] 10 Computer system, 100 Information processing device, 1000 Network management device, 101 Real-time application, 102 General-purpose application, 103 Real-time container, 104 General-purpose container, 105 Virtualization software, 106 Processor, 107 Memory, 108 I / O device, 109 Graphics device, 110 Network device, 111 Transmission management buffer, 120 Bus, 200 Time synchronization function management unit, 210 Master container selection unit, 220 Master server selection unit, 230 Master setting management unit, 240 List of devices excluded from migration, 300 Device sharing management unit, 310 Bandwidth usage data acquisition unit, 320 Shared configuration identification unit, 330 Missing data determination unit, 340 Communication cycle change unit, 350 Bandwidth usage rate list, 360 Shareable container list, 400 TSN data transmission management unit, 410 Data transmission detection unit, 420 430 Transmission data storage unit, 500 Transmission execution unit, 510 Resource allocation management unit, 520 Resource utilization rate collection unit, 530 Evaluation parameter identification unit, 540 Allocation identification unit, 550 Resource utilization rate list, 550 Collected data.
Claims
1. A network management device comprising: a master container selection unit that selects one of the containers belonging to a selected clock domain as a time synchronization master container, with each of the multiple clock domains to which multiple containers operating on multiple information processing devices constituting a computer system belong as the selection target; and a master server selection unit that selects an information processing device from among the multiple information processing devices whose clock specifications meet the criteria as a time synchronization master server on which the time synchronization master containers selected by the master container selection unit for each of the multiple clock domains are located.
2. The network management device further comprises a master setting management unit that determines whether the time synchronization master containers selected for each of the plurality of clock domains can be placed on the time synchronization master server selected by the master server selection unit, and the master server selection unit further selects a new time synchronization master server to place the time synchronization master containers from the plurality of information processing devices other than the time synchronization master server until the master setting management unit determines that it can be placed there.
3. The network management device according to claim 2, wherein the master server selection unit generates a time synchronization slave container that performs time synchronization among a plurality of time synchronization master servers.
4. The network management device according to claim 2 or 3, wherein the master setting management unit determines whether or not the time synchronization master container can be placed on the time synchronization master server based on the utilization rate of hardware resources when the computer system is in operation.
5. The network management device according to claim 1, further comprising a master setting management unit that controls the migration of the time synchronization master container from the time synchronization master server to an information processing device other than the time synchronization master server.
6. The network management device according to any one of claims 1 to 5, further comprising a shared configuration identification unit that identifies a combination of containers that can share a network device provided by any of the multiple information processing devices based on network bandwidth usage information of each of the multiple containers, and identifies an information processing device to which each of the multiple containers is located according to the identified combination.
7. The network management device according to claim 6, wherein the shared configuration identification unit identifies an information processing device to which each of the plurality of containers is located in order to satisfy the requirements regarding hardware resources.
8. The network management device according to claim 6 or 7, further comprising a loss determination unit that determines whether or not data loss occurs in transmitted data when the plurality of containers are placed in the information processing device at the destination identified by the shared configuration identification unit.
9. The network management device further includes a communication cycle modification unit that modifies the communication cycle of at least some of the containers among the plurality of containers when the data loss determination unit determines that data loss has occurred, and the data loss determination unit determines whether or not data loss occurs in the transmitted data when the communication cycle modified by the communication cycle modification unit is applied.
10. The network management device according to any one of claims 1 to 9, wherein when an abnormality occurs in the time synchronization master container for the clock domain to be processed among the plurality of clock domains, if there is a container belonging to the clock domain to be processed that is located on the time synchronization master server, the master container selection unit selects that container as the new time synchronization master container, and if there is no container belonging to the clock domain to be processed that is located on the time synchronization master server, selects any container from the containers belonging to the clock domain to be processed as the new time synchronization master container and places it on the time synchronization master server.
11. The network management device according to any one of claims 1 to 10, wherein when a new information processing device is added to the computer system, the master server selection unit selects the new information processing device as the new time synchronization master server in place of the time synchronization master server if the new information processing device has better clock specifications than the time synchronization master server.
12. The network management device according to claim 2, wherein when a new clock domain is added, the master container selection unit selects one of the containers belonging to the new clock domain as a time synchronization master container, and the master setting management unit determines whether or not the time synchronization master container for the new clock domain can be placed on the time synchronization master server.
13. A network management method comprising: a computer selecting multiple clock domains to which multiple containers operating on multiple information processing devices constituting a computer system belong as selection targets, selecting one of the containers belonging to the selected clock domain as a time synchronization master container, and the computer selecting an information processing device among the multiple information processing devices whose clock specifications meet the criteria as a time synchronization master server on which the time synchronization master container selected for each of the multiple clock domains is located.
14. A network management program that causes a computer to function as a network management device, performing a master container selection process in which a master container is selected as a time synchronization master container, and a master server selection process in which an information processing device among the multiple information processing devices whose clock specifications meet the criteria is selected as a time synchronization master server where the time synchronization master container selected for each of the multiple clock domains by the master container selection process is located.