System for simulating operation equipment in a production environment

Abstract

A system and devices for generating and consuming three-dimensional immersive environments of one or more facilities (e.g., shipping yards, assembly plants, processing plants, warehouses, distribution centers, ports, yards, transports, and the like). A simulation system may be configured to generate a full three-dimensional (3D) virtual environment or mixed reality environment that replicates the physical layout of the facility as well as the operations of the facility. The simulation system may also be utilized to simulate potential operations of the facility, such as with new or modified settings, configurations, arrangements, assignments, equipment, personnel, and the like.

Description

SYSTEM FOR SIMULATING OPERATION EQUIPMENT IN A PRODUCTION ENVIRONMENTCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to U.S. Provisional Application No. 63 / 713,186, filed on October 29, 2024 and entitled “System for simulating operation equipment in a production environment,” the entirety of which is incorporated herein by reference.BACKGROUND

[0002] Facilities, such as shipping yards, assembly plants, processing plants, warehouses, distribution centers, ports, yards, transports, and the like store and process vast quantities of assets, inventory, and items over various periods of time using various arrangements and configurations of equipment. In some situations, introduction of new equipment or adjustments to the configurations, settings, or arrangement of the equipment can be expensive and time consuming as the plant is idled for a period of time. In some specific cases, the new equipment or the adjustments may cause further costs and delays as the equipment and / or adjustments do not perform as expected.BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The detailed description is described with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical components or features.

[0004] FIG. 1 is an example block diagram of a simulation system and platform for evaluating equipment and planned operations at a facility according to some implementations.

[0005] FIG. 2 is an example block diagram of a simulation system for evaluating equipment and planned operations at a facility according to some implementations.

[0006] FIG. 3 is an example block diagram of a simulation system for evaluating equipment and planned operations at a facility according to some implementations.

[0007] FIG. 4 is a flow diagram illustrating an example process associated with generating three-dimensional scenes associated with a facility according to some implementations.

[0008] FIG. 5 is a flow diagram illustrating an example process associated with a simulation system operating at a facility according to some implementations.

[0009] FIG. 6 is another flow diagram illustrating an example process associated with simulation system operating at a facility according to some implementations.

[0010] FIG. 7 is an example simulation system that may implement the techniques described herein according to some implementations.DETAILED DESCRIPTION

[0011] Discussed herein are systems and devices for simulating operations of equipment operating to perform tasks at one or more facilities (e.g., shipping yards, assembly plants, processing plants, warehouses, distribution centers, ports, yards, transports, and the like). In some cases, the simulation system may be configured to generate a full three-dimensional (3D) virtual environment or mixed reality environment that replicates the physical layout of the facility as well as the operations of the facility. For example, the simulation system may receive sensor data (including 3D image data of the facility) captured by sensors associated with the facility (e.g., on equipment, personnel, fixed through the facility, and the like). The simulation system may also receive operational data, such as orders, throughput, equipment data (e.g., maintenance, age, status, operations, assigned tasks, arrangements, settings, configurations, and the like), personnel data (e.g., employment records and history, age, proficiency, licenses, performance at various tasks or with respect to operation of various equipment, and the like), incoming and outgoing shipment data (e.g., schedules, expected arrivals and departures, on time task performance records, and the like).

[0012] In some cases, the simulation system may then generate a 3D virtual environment of the facility including equipment, personnel, inventory', operations, and the like that may represent past operations of the facility (e.g.. that may replicate the actual operations of the facility) and is viewable via 3D immersion technology (3D virtual / mixed reality headsets) in the 3D virtual environment or with two-dimensional (2D) displays. In this manner, a 3D immersive environment or record of the operationsof the facility may be viewed, reviewed, referenced, and / or otherwise accessed by one or more users of the simulation system.

[0013] In some implementations, the 3D virtual environment of the facility may also be utilized to simulate potential operations of the facility. For example, the simulation system may allow a user to modify settings, configurations, arrangement, assignments, and the like of equipment, personnel, inventory, storage racks, bins, or areas, and the like with respect to the operations of the facility. The simulation system may then generate a simulation of the operations of the overall facility, a particular assembly or production line, or the like as an instance of the 3D virtual environment over a selected period of time. As one illustrative example, a user may select a specific period of time from the historical simulation of actual operations of the facility and apply one or more changes or series of changes to the operations (e.g., increasing speeds of a conveyor belt, modified assignment of equipment or personnel, increasing a shift by fifteen minutes, and / or the like) and the simulation system may generate an instance of the 3D virtual environment over the selected period of time reflecting the changes.

[0014] In another implementations, the 3D virtual environment of the facility’ may also be utilized to simulate potential operations of the facility including the introduction of new or potential equipment, processes, methodologies, personnel, and / or the like. For example, the simulation system may allow selection of a virtual version of a new piece of equipment, such as a new conveyor belt. In some cases, the simulation system may receive details or data associated with new available equipment for use in one or more facilities and generate a virtual version of the equipment (e.g., a selectable object model that may be inserted into a 3D virtual environment or scene). In this manner, the dimensions, characteristics, performance metrics, and the like may be simulated with respect to the operations of a specific facility with specific employees, inventor}', and other companion equipment. As discussed above, a user may also select a period of time for the simulation to execute over, such as a specific period of time from the historical simulation of actual operations of the facility. In other cases, the simulation system may allow a user to define a simulation instance with select resources (e.g., equipment, personnel, processes, and / or the like) from a list of available resources to generate a synthetic scene of potential operations of the facility given the user inputs.

[0015] In some cases, the simulation system may allow a user to place, move, rotate, or otherwise arrange the new equipment object model within the 3D scene orenvironment of the facility as well as to arrange other object models of the existing resources within the 3D scene prior to initiating the simulation instance.

[0016] In another example, the simulation system may generate multiple (e.g., tens, hundreds, thousands, tens of thousands, or the like) instances of the simulation introducing the object model of the new equipment into the 3D scene of the facility over the selected period of time with various different configurations of the existing facility resources. The simulation system may then execute each simulation instance to generate a plurality of results data each of which represents the outcome or performance of the facility' utilizing the selected resource combination and the new equipment. The system may also run multiple instances of various new equipment using different object models for each unit available for a given type of equipment (e.g.. forklift, conveyor, crane, robotic arms, robotic pickers, robotic inventory retrieval units, palletization robots, and the like). In this example, the simulation system may determine compatibility of each object model available for the select object type with the facility and, for each compatible model (e.g.. new equipment), generate multiple simulation instances as discussed above.

[0017] Once the simulation instances are generated for each of the compatible obj ect models or new equipment, the simulation system may generate a report including the result data for each simulation instance for each object model type or new equipment. The system may also aggregate the result data for each object model type or new equipment as well as provide a ranking or comparison of the performance for each object model type or new equipment. In some cases, the simulation system may present a cost to output analysis for each object model type or new equipment and rank based on the costs to output analysis.

[0018] In addition to the report, the simulation system may render a 3D scene or environment that may be accessible to a user to watch, via a 2D or 3D display, the operations of one or more particular simulation instance. In this manner, a user can view at various speeds (slow motion, real-time, or fast forward) the operations of the facility with the object model type or new equipment with respect to the existing resources. Accordingly, an operations specialist may be able to adjust additional resources, configurations, positioning and the like with respect to the object model type or new equipment to further test and / or improve the operations of the facility with the potential new equipment.

[0019] In the above example, a new object model for new equipment was stimulated by the simulation system. However, it should be understood that the simulation system may also generate and execute simulation instances for new processes, new methodologies, new personnel, adjusted existing resources, as well as other resources and the like.

[0020] In various examples, the simulation system may utilize one or more machine learning models and / or networks to assist in generating the simulation instances, the object models, executing the simulation, generating the 3D environments or scenes, aggregating, sorting, and / or ranking the result data, and the like. As an example, the simulation system may be trained on historical data of the operations of the facility' as well as via the synthetic simulation instances. For instance, an operator may flag various simulation instances as training data following a review of the output of the simulation system. These executed simulation instances may then be input into the one or more machine learning models as training data.

[0021] In one specific example, the simulation system may be configured to utilize the output of the one or more machine learning models and / or networks to select, hire, and / or order a new resource (e g., equipment, personnel, or the like) on behalf of the facility without human input based on the results of the simulation instances. In another specific example, the simulation system may be configured to utilize the output of the one or more machine learning models and / or networks and the simulation instances to adjust a configuration of the existing resources (e g., settings, configurations, assignments, operational hours, tasks, process flows, and / or the like) on behalf of the facility without human input based on the results of the simulation instances.

[0022] In some examples discussed herein, the sensors may be internet of things (loT) computing devices that may be equipped with various sensor and / or image capture technologies, and configured to capture, parse, and identify vehicle and container information from the exterior of vehicles, containers, pallets, and the like. The vehicle and container information may include shipping documents, such as BOL (Bill of Lading), packing list, container identifiers, chassis identifiers, vehicle identifiers, and the like. The loT computing devices may also capture, parse, and identify driver information in various formats, such a driver licenses, driver’s identification papers, facial features and recognition, and the like.

[0023] As discussed above, the system may include multiple loT devices at various locations as well as cloud-based services, such as cloud-based data processing. One ormore loT computing device(s) may be installed at entry and / or exit points of a facility. The loT computing devices may include a smart network video recorder (NVR) or other type of EDGE computing device. Each loT device may also be equipped with sensors and / or image capture devices usable at night or during the day. The sensors may be weather agnostic (e.g., may operate in foggy, rainy, or snowy conditions), such as via infrared image systems, radar based image systems, LIDAR based image systems, SIWIR based image systems. Muon based image systems, radio wave based image systems, and / or the like. The loT computing devices and / or the cloud-based services may also be equipped with models and instructions to capture, parse, identify, and extract information from the vehicles, containers, and / or various documents associated with the logistics and shipping industry. For example, the loT computing devices and / or the cloud-based services may be configured to perform segmentation, classification, attribute detection, recognition, document data extraction, and the like. In some cases, the loT computing devices and / or an associated cloud-based service may utilize machine learning and / or deep learning models to perform the various tasks and operations.

[0024] In some cases, since the sensor data received may be from different sources or types of sensors at different ranges and generalities, the loT computing devices may perform a data normalization using techniques such as threshold-based data normalization and machine learning algorithms to identify the driver, vehicle, or container. It should be understood that the system may utilize different weighted averages or thresholds based on the data source (e.g., sensor type, location, distance, and position), the current weather (e.g., sunny, rainy, snowy, or foggy), and time of day when performing data normalization. In some cases, machine learning algorithms may also be applied to remove the distortion from images caused by rain, dust, sand, fog, and the like as well as to brighten the sensor and / or images shot in low-light or dark conditions.

[0025] As described herein, the machine learned models may be generated using various machine learning techniques. For example, the models may be generated using one or more neural network(s). A neural network may be a biologically inspired algorithm or technique which passes input data (e.g., image and sensor data captured by the loT computing devices) through a series of connected layers to produce an output or learned inference. Each layer in a neural network can also comprise another neural network or can comprise any number of layers (whether convolutional or not). As canbe understood in the context of this disclosure, a neural network can utilize machine learning, which can refer to a broad class of such techniques in which an output is generated based on learned parameters.

[0026] As an illustrative example, one or more neural network(s) may generate any number of learned inferences or heads from the captured sensor and / or image data. In some cases, the neural network may be a trained network architecture that is end-to- end. In one example, the machine learned models may include segmenting and / or classifying extracted deep convolutional features of the sensor and / or image data into semantic data. In some cases, appropriate truth outputs of the model in the form of semantic per-pixel classifications (e.g., vehicle identifier, container identifier, driver identifier, and the like).

[0027] Although discussed in the context of neural networks, any type of machine learning can be used consistent with this disclosure. For example, machine learning algorithms can include, but are not limited to, regression algorithms (e g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), locally estimated scatterplot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net, least-angle regression (LARS)), decisions tree algorithms (e.g., classification and regression tree (CART), iterative dichotomiser 3 (ID3), Chi-squared automatic interaction detection (CHAID), decision stump, conditional decision trees), Bayesian algorithms (e g., naive Bayes, Gaussian naive Bayes, multinomial naive Bayes, average one-dependence estimators (AODE), Bayesian belief network (BNN), Bayesian networks), clustering algorithms (e.g., k- means, k-medians, expectation maximization (EM), hierarchical clustering), association rule learning algorithms (e.g., perceptron, back-propagation, hopfield network, Radial Basis Function Network (RBFN)), deep learning algorithms (e.g., Deep Boltzmann Machine (DBM), Deep Belief Networks (DBN), Convolutional Neural Network (CNN), Stacked Auto-Encoders), Dimensionality Reduction Algorithms (e.g., Principal Component Analysis (PCA). Principal Component Regression (PCR), Partial Least Squares Regression (PLSR), Sammon Mapping, Multidimensional Scaling (MDS), Projection Pursuit, Linear Discriminant Analysis (LDA), Mixture Discriminant Analysis (MDA), Quadratic Discriminant Analysis (QDA). Flexible Discriminant Analysis (FDA)), Ensemble Algorithms (e.g., Boosting, Bootstrapped Aggregation (Bagging), AdaBoost, Stacked Generalization(blending), Gradient Boosting Machines (GBM), Gradient Boosted Regression Trees (GBRT), Random Forest), SVM (support vector machine), supervised learning, unsupervised learning, semi-supervised learning, etc. Additional examples of architectures include neural networks such as ResNet50, ResNetlOl, VGG, DenseNet, PointNet, and the like. In some cases, the system may also apply Gaussian blurs, Bayes Functions, color analyzing or processing techniques and / or a combination thereof.

[0028] FIG. 1 is an example block diagram 100 of a simulation system and platform 102 for evaluating equipment and planned operations at a facility according to some implementations. As discussed herein, the simulation system 102 may be configured to monitor a facility such as a shipping yard, assembly plant, processing plant, warehouse, distribution center, port, yard, transport, and the like. In some cases, the simulation system 102 may be configured to generate a 3D virtual environment or mixed reality environment or scene that replicates the physical layout of the facility as well as the operations of the facility over various period of time.

[0029] For instance, the simulation system 102 may be configured to capture sensor data 104 from various sensor systems 106 throughout the facility. The sensor systems 106 may be mounted or fixed throughout the facility, associated with faci 1 i ty equipment or personnel, and / or associated w ith delivery vehicles that pickup and drop off assets, items, and / or inventory at the faci li ty. The simulation system 102 may then generate a 3D scene 108 of the facility representing the operations of the facility captured by the sensor systems 106 in a manner that may be viewed or accessed by one or more users via one or more display systems 110. Accordingly, a remote personnel or third-party (e.g., a customer, inspector, government official, potential customer, or the like) mayview and evaluate the operations of the facility- without requiring an in-person visit.

[0030] In addition to generating one or more 3D scenes 108 of the facility, the simulation system 102 may also be configured to allow a user to virtually test or assess changes to the operations and process of the facility- based on historical 3D virtual scenes of past performances with one or more modified settings, configurations, arrangements, and / or the like. For instance, a user may provide simulation configuration data 112 to the simulation system 102 that may be used to modify an existing 3D scene 108. The simulation system 102 may then generate a simulation instance based at least in part on the simulation configuration data 112 and one or more historical 3D scene 108.

[0031] As one specific example, the simulation system 102 may provide the simulation configuration data 112, selected sensor data 104, and selected 3D scenes 108 to one or more machine learning models 114. The one or more machine learning models 114 may then generate and provide to the simulation system 102 simulation instance data 118 that may be utilized by the simulation sy stem 102 to execute a simulation of the facility including any changes to a 3D scene 108 based on the simulation configuration data 112. The simulation system 102 may then execute the simulation instance to generate result data 116 as well as an additional 3D scene 108 that represents the operations of the facility given the simulation configuration data 112 and the historical data associated with the facility as an accessible, immersible, and viewable environment. In some cases, the 3D scenes 108 generated by the execution of a simulation instance may be considered synthetic 3D scenes 108 that may be further used to train the simulation system 102 and / or the machine learning models 114 associated therewith.

[0032] The simulation system 102 may also generate the result data 116 for each simulation instance executed. In some cases, the one or more machine learning models 114 may generate a plurality of simulation instances represented by the simulation instance data 118 which may cause the system 102 to generate result data 116 for each simulation instance (such as the output quality metric or quantity metric associated with each instance of the changed operation or process). In some cases, the simulation system 102 may generate, aggregate, sort, rank, and / or recommend modified operations at the facility’ based on the result data 116 associated with the plurality of simulation instances. In some cases, the one or more machine learning models 114 may be configured to generate the recommended modified operations based on an input of the result data 116.

[0033] In some cases, the simulation system 102 may also generate reports or report data 120 representing the aggregated, sorted, and / or ranked result data as well as the recommended modified operations. For example, the report data 120 of the plurality' of executions of the simulation instances may be presented in a manner that is easily digestible or reviewable by one or more users, such as via a display system 110. Together, the report data 120 and the 3D scenes 108 of one or more highlighted simulation instance execution may be accessed and assessed by a user to determine if a change to the operation of the facility would be beneficial to the overall operations of the facility. In this manner, unlike conventional evaluations or reports, the systemdiscussed herein allows a user to review the operations via the 3D scene to personally evaluate the operations even when not on site at the facility or at a later time. In some cases, the 3D scenes 108 may be reviewed by groups of users, each of which may be immersed or virtually present within the 3D scene 108 even when the users are at remote physical locations from each other and / or the facility.

[0034] The simulation system 102 may also be configured to allow a user to virtually introduce new equipment, personnel, and the like into the virtual 3D scenes 108 to evaluate the performance or throughput of the facility with the applicable or selected new equipment, personnel, and / or the like. For example, the simulation system 102 may be configured to receive object model data 122 from a third-party system 124 (such as a seller of the equipment) an / or generate the object model data 122 from schematics or available specifications of the new' equipment or personnel via the one or more machine learning models 114 trained on data associated with facility equipment, vehicles, personnel, and / or the like.

[0035] The simulation system 102 may also allow a user to select, replace existing equipment, insert, or otherwise arrange the object models of the new equipment within a 3D scene 108. The user may also select various configurations for the equipment, personnel, or other resource represented by the 3D model for use in generating the simulation instance data 118 associated with the resource being evaluated or considered for introduction into the facility. For example, the simulation system 102 may then generate one or more simulation instances based at least in part on the object model data 122, the simulation configuration data 112 received from the user, and one or more historical 3D scenes 108, as discussed above.

[0036] As one specific example, the simulation system 102 may provide the object model data 122, the simulation configuration data 112, selected sensor data 104, and selected 3D scenes 1 18 to one or more machine learning models 114. The one or more machine learning models 114 may then generate and provide to the simulation system 102 simulation instance data 118 that may be utilized by the simulation system 102 to execute a simulation of the facility including the introduction of the new resource represented by the object model data 122 into one or more selected 3D scenes 108 based at least in part on the simulation configuration data 112.

[0037] The simulation system 102 may then execute the simulation instance to generate result data 116 as well as an additional 3D scene 108 that represents the operations of the facility given the object model data 122, the simulation configurationdata 112, and the historical data associated with the facility as an accessible, immersible, and viewable environment. As discussed above, in some cases, the 3D scenes 108 generated by the execution of a simulation instance may be considered synthetic 3D scenes 108 that may be further used to train the simulation system 102 and / or the machine learning models 114 associated therewith.

[0038] The simulation system 102 may also generate the result data 116 for each simulation instance executed with the new resource or potential new resources when more than one resource is being considered. As discussed above, in some implementations, the one or more machine learning models 114 may generate a plurality of simulation instances represented by the simulation instance data 118 which may cause the system 102 to generate result data 116 for each simulation instance (such as the output quality metric or quantity metric associated with each instance of the changed operation or process). In some cases, the simulation system 102 may generate, aggregate, sort, rank, and / or recommend modified operations at the facility based on the result data 116 associated with the plurality of simulation instances. In some cases, the one or more machine learning models 114 may be configured to generate the recommend modified operations based on an input of the result data 116.

[0039] In some cases, the simulation system 102 may also generate reports or report data 120 representing the aggregated, sorted, and / or ranked result data as well as the recommend modified operations. For example, the report data 120 of the plurality of executions of the simulation instances may be present in a manner that is easily digestible or reviewable by one or more users, such as via a display system 110. For example, the report data 120 may include atop three performing new resources together with access links for consuming one or more 3D virtual scenes 108 of the new resources in use. Together, the report data 120 and the 3D scenes 108 of one or more highlighted simulation instance execution may be accessed and assessed by a user to determine if introduction of the new resource to the operation of the facility would be beneficial to the overall operations of the facility.

[0040] In the current example, the data and 3D scenes as well as other data may be transmitted between various systems using networks, generally indicated by 126-132. The networks 126-132 may be any type of network that facilitates compunction between one or more systems and may include one or more cellular networks, radio, WiFi networks, short-range or near-field networks, infrared signals, local area networks, wide area networks, the internet, and so forth. In the current example, eachnetwork 126-132 is shown as a separate network but it should be understood that two or more of the networks may be combined or the same.

[0041] FIG. 2 is an example block diagram of a simulation system 200 for evaluating equipment and planned operations at a facility according to some implementations. As discussed herein, the simulation system 200 may be configured to monitor a facility such as a shipping yard, assembly plant, processing plant, warehouse, distribution center, port, yard, transport, and the like. In some cases, the simulation system 200 may be configured to generate a 3D virtual environment or mixed reality environment or scene that replicates the physical layout of the facility as well as the operations of the facility over various period of time.

[0042] In the current example, the simulation system 200 may include a configuration system 202 or component that may be configured to receive the sensor data 210, user inputs 212 (e.g., scenario settings, configurations, and the like), as well as third party data, such as object model data 214. In some cases, the configuration system 202 may generate instructions for the one or more machine learning models and / or networks 204 to generate simulation data 216 based at least in part on the sensor data 210, user inputs 212, object model data 214 as well as one or more selected historical 3D scenes being modified with respect to the simulation data 216. In some cases, the configuration system 202 may generate the simulation data 216 for the simulation instances 218. In other cases, such as the illustrated example, the configuration system 202 may provide the sensor data 210, user inputs 212, object model data 214, historical 3D scenes and / or additional instructions for the machine learning models 204 to the machine learning models 204 and receive as an output of the one or more machine learning models the simulation data 216 associated with the simulation instance 218. In some specific implementations, the configuration system 202 may receive the simulation data 216 (including object models, object settings, possible arrangements, process data, and the like) from the one or more machine learning models 204 and the configuration system 202 may generate the simulation instances 218 based on the simulation data 216, such as selecting object models representing potential resources and currently available resources for each simulation instance 218. In this manner, the configuration system 202 may generate simulation instances 218 to evaluate potential equipment with a plurality of different operators.

[0043] The execution system 206 may receive the simulation instances 218 from the configuration system 202 and execute each instance 218 to generate results data 220for each of the simulation instances 218 executed. In this manner, the execution system 206 may generate the result data 220 as well as one or more 3D scene 222 for each of the simulation instances 218 executed.

[0044] In some implementations, the result data 220 may be provided to the one or more machine learning models 204 such that the one or more machine learning models 204 may perform operations to process the result data 220 and generate aggregated data 224 and / or report data 226. For example, the one or more machine learning models 204 may be configured to detect trends and / or evaluate performance (such as facility performance and / or resource performance, or sets / groups of resources performances, and the like). In some cases, the one or more machine learning models 204 may be configured to rank potential equipment for use in the facility. In some cases, the one or more machine learning models 204 may be configured to detect safety concerns related to particular operations, processes, resource arrangements / pairings, resources placements, and the like.

[0045] In the current example, the one or more machine learning models 204 are illustrated as the same models for the generation of the simulation data 216 and the aggregation data 224, however, it should be understood that the one or more machine learning models 204 may be different sets of models trained on different data including different types of data with different training processes. Likewise, the one or more machine learning models 204 may include multiple sets of machine learning models configured to receive as an input an output of an upstream model or network. In other cases, the one or more machine learning models may be a network that is trained end to end with or without multiple heads.

[0046] In the illustrated example, the simulation system 202 may include a user interface system 208. The user interface system 208 may be integrated into the simulation system 202 or a remote system such as a user device, communicatively coupled to the simulation system 202. For example, the user interface system 208 may include user hardware hosting a downloadable application that interfaces with the simulation system 202 for consumption of the 3D scenes 222 and / or the report data 226.

[0047] FIG. 3 is an example block diagram of a simulation system 300 for evaluating equipment and planned operations at a facility according to some implementations. As discussed herein, the simulation system 300 may be configured to monitor a facility such as a shipping yard, assembly plant, processing plant, warehouse, distribution center, port, yard, transport, and the like. In some cases, the simulationsystem 300 may be configured to generate a 3D virtual environment or mixed reality environment or scene that replicates the physical layout of the facility as well as the operations of the facility over various periods of time.

[0048] In the current example, one or more machine learning models or networks 302 of the simulation system 300 may receive data for use in generating simulation data 304. For example, the one or more machine learning models 302 may receive resource data 306 (e.g., data associated wi th equipment, vehicles, personal, and / or other assets of the facility), sensor data 308 (e.g., capture of the operations of the facility during one or more period of time), process data 310 (e.g., data representing best practices, known procedures, operations, methodologies, and the like implemented at the facility, and / or the like). 3D scene data 312 (e.g., 3D scenes generated from prior period of time associated with the facility, generated from other simulation instances, and / or the like), object model data 314 (e.g., representing potential resources for the facility and / or the like), inventor}' data 314 (e.g., data representing the current inventory or potential inventory of the facility), and / or the like. The one or more machine learning models 302 may then generate, based at least in part on the input, the simulation data 304 for evaluating changes to the facility’ operations and / or introduction of potential resources, as discussed herein.

[0049] In the current example, the configuration system 318 may receive the simulation data 304 and generate one or more simulation instances 320 for execution by the execution system 322 as discussed herein. The execution system 322 may be configured to execute the simulation instances 320 to generate 3D scenes or scene data 324 and result data 326. In some cases, the result data 326 may be further processed such as aggregated, evaluated, ranked, and the like by one or more machine learning models, the configuration system 318, and / or another system of the simulation system 300.

[0050] FIGS. 4-6 are flow diagrams illustrating example processes associated with the simulation system discussed herein. The processes are illustrated as a collection of blocks in a logical flow diagram, which represent a sequence of operations, some or all of which can be implemented in hardware, software, or a combination thereof. In the context of software, the blocks represent computer-executable instructions stored on one or more computer-readable media that, which when executed by one or more processor(s), perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, encryption, deciphering.compressing, recording, data structures and the like that perform particular functions or implement particular abstract data types.

[0051] The order in which the operations are described should not be construed as a limitation. Any number of the described blocks can be combined in any order and / or in parallel to implement the processes, or alternative processes, and not all of the blocks need be executed. For discussion purposes, the processes herein are described with reference to the frameworks, architectures and environments described in the examples herein, although the processes may be implemented in a wide variety of other frameworks, architectures or environments.

[0052] FIG. 4 is a flow diagram illustrating an example process 400 associated with generating three-dimensional scenes associated with a facility according to some implementations. As discussed herein, a simulation system may be configured to monitor a facility such as a shipping yard, assembly plant, processing plant, warehouse, distribution center, port, yard, transport, and the like. In some cases, the simulation system may be configured to generate a 3D virtual scene or environment that replicates the physical layout of the facility as well as the operations of the facility over various period of times. In some cases, the simulation system may also generate metrics or data associated with the facility that may be overlay ed onto the 3D scenes to provide insights and understanding of the facility (such as inventory counts, safety metrics, resource maintenance and repair statuses, and the like).

[0053] At 402, the simulation system may receive sensor data associated with a facility over a period of time. For example, the sensor data may be received from one or more sensors positioned throughout the facility, such as at a gate check-in or checkout locations, loading areas, unloading areas, assembly areas, storage areas, packaging areas, and the like. The sensor data may also be received from one or more sensors systems associated with facility resources (e.g., equipment, machinery, personnel, and the like), third-party7systems (e.g., shipping and transport systems, government systems, transport vehicles, ordering systems, point-of-sales devices, and the like), and the like. In some cases, the sensor data may include LIDAR data, SIWIR data, red- green-blue image data, thermal data, Muon data, radio wave data, weight data, infrared data, and the like.

[0054] At 404, the simulation system may generate, based at least in part on the sensor data, a 3D scene representing the facility over the period of time. For example, the system may generate a 3D immersive environment that may be consumed by a uservia a 2D display and / or a 3D immersive display based on the sensor data associated with the facility’ over the period of time. In this manner, a user (e.g., such as a customer, potential customer, employee, third-party consultant, and the like) may view the operations of the facility (either independently or as a group within a shared scene) to evaluate the operations of the facility without having to be physically present on-site. In some cases, the users consuming the content may be able to provide or attach comments (e.g., text, audio, visual data, or the like) to the 3D scene so that the user and / or others may interact with the comments while they are consuming the 3D scene at a later time.

[0055] At 406, the simulation system may determine, based at least in part on the sensor data, one or more metrics associated with the facility during the period of time. For example, the simulation system may determine a throughput, inventory counts, operational metrics (e.g., time of use, idle time, and the like associated with one or more resource, and the like), resource pairing or collaboration metrics, performance metrics, safety metrics (e.g., comparison of approve operational procedures v. actual procedures, records of safety violations, injuries, or other incidents, and the like), any other incident or concerns (e g., equipment traffic blockages in an aisle, lost inventory, and the like), and the like.

[0056] At 408, the simulation system may determine, based at least in part on the one or more metrics and historical data associated with the facility, one or more recommendations associated with the facility. For example, the system may recommend additional safety training, changes in equipment routes, settings, configuration, parings or combinations, assignments, and the like.

[0057] At 410, the simulation system may augment the 3D scene with the one or more metrics and / or the one or more recommendations. For example, the system may determine changes or modifications to the operations of the facility that may result in improvements to efficiency, safety, idle time, and the like based at least in part on the sensor data and / or historical data, best practices, and procedures, and / or user selected goals / initiatives.

[0058] At 412, the simulation system may display the 3D scene to a user. For example, a user may consume the 3D scene by interacting with the scene via a 3D immersive display system that allows the user to both view and interface (e.g., modify) the 3D scene as if the user was present on-site during the period of time.

[0059] FIG. 5 is a flow diagram illustrating an example process 500 associated with simulation system operating at a facility according to some implementations. As discussed herein, a simulation system may be configured to monitor a facility such as a shipping yard, assembly plant, processing plant, warehouse, distribution center, port, yard, transport, and the like. In some cases, the simulation system may be configured to generate a 3D virtual scene or environment that replicates the physical layout of the facility as well as the operations of the facility over various period of times. In some cases, the simulation system may also allow a user to modify the operations of the facility (e.g., resources, arrangements, operations, procedures, methodologies, and the like) and to generate one or more additional 3D scenes simulation the facility during the period of time implementing the user modifications. In some cases, the simulation system may determine metrics and / or recommendations associated with the modification based on the results of the execution of the simulations in order ot improve the overall operations of the facility.

[0060] At 502, the simulation system may receive sensor data associated with a facility over a period of time. For example, the sensor data may be received from one or more sensors positioned throughout the facility, such as at a gate check-in or checkout locations, loading areas, unloading areas, assembly areas, storage areas, packaging areas, and the like. The sensor data may also be received from one or more sensors systems associated with facility resources (e.g., equipment, machinery, personnel, and the like), third-party systems (e.g., shipping and transport systems, government systems, transport vehicles, ordering systems, point-of-sales devices, and the like), and the like. In some cases, the sensor data may include LIDAR data, SIWIR data, red- green-blue image data, thermal data, Muon data, radio wave data, weight data, infrared data, and the like.

[0061] At 504, the simulation system may receive object model data associated with one or more resources. For example, the simulation system may generate an object model or object model data for each resource operating within the facility during the period of time. In some cases, the object model data may be received from third-party systems, such as when the resource is equipment purchased from a vendor that allows for remote equipment testing or evaluation.

[0062] At 506, the simulation system may receive one or more user inputs modifying at least one resource, process, and / or operation associated with the facility. For example, the user may swap resources, change resource settings or configuration.change processes or methodologies, and the like. In some cases, the user may input a range of settings or configurations, arrangements, resource swaps, and the like to test a variety of different combinations within the facility without having to physically relocate resources or experience delays due to ill-advised choices.

[0063] At 508, the simulation system may generate, based at least in part on the sensor data, the object model data, and the user inputs, simulation data associated with the facility and, at 510, the simulation system may generate, based at least in part on the simulation data, one or more simulation instances associated with the faculty. For example, the simulation system may generate a simulation instance for each of the variety of different combinations that the user desires to evaluate within the facility.

[0064] At 512, the simulation system may generate, based at least in part an execution of individual ones of the one or more simulation instances, result data and a 3D scene. For example, the simulation system may execute each simulation instance to produce result data and a 3D scene for each individual simulation instance.

[0065] At 514, the simulation system may determine, based at least in part on the result data, one or more recommendations associated with the facility. For example, the system may rank, score, or otherwise determine metrics associated with each option to modify the facility operations and determine a recommendation based on the rankings, scores, and metrics. For example, the system may determine a modification resulting in the largest increase in throughput for the facility for a most common process and then recommend to a user that the facility implement the modification during physical operations.

[0066] At 516, the simulation system may augment the 3D scene with the result data and / or the one or more recommendations. For example, the system may determine changes or modifications to the operations of the facility that may result in improvements to efficiency, safety7, idle time, and the like based at least in part on the sensor data and / or historical data, best practices, and procedures, and / or user selected goals / initiatives.

[0067] At 518. the simulation system may display the 3D scene to a user. For example, a user may consume the 3D scene by interacting with the scene via a 3D immersive display system that allows the user to both view and interface (e.g., modify) the 3D scene as if the user was present on-site during the period of time.

[0068] FIG. 6 is another flow diagram illustrating an example process 600 associated with simulation system operating at a facility' according to someimplementations. As discussed herein, a simulation system may be configured to monitor a facility such as a shipping yard, assembly plant, processing plant, warehouse, distribution center, port, yard, transport, and the like. In some cases, the simulation system may be configured to generate a 3D virtual scene or environment that replicates the physical layout of the facility' as well as the operations of the facility over various period of times. In some cases, the simulation system may also allow a user to modify the operations of the facility (e.g., resources, arrangements, operations, procedures, methodologies, and the like) and to generate one or more additional 3D scenes simulation the facility during the period of time implementing the user modifications. In some cases, the simulation system may determine metrics and / or recommendations associated with the modification based on the results of the execution of the simulations in order ot improve the overall operations of the facility.

[0069] At 602, the simulation system may receive sensor data associated with a facility over a period of time. For example, the sensor data may be received from one or more sensors positioned throughout the facility, such as at a gate check-in or checkout locations, loading areas, unloading areas, assembly areas, storage areas, packaging areas, and the like. The sensor data may also be received from one or more sensors systems associated with facility resources (e.g., equipment, machinery, personnel, and the like), third-party' systems (e.g., shipping and transport systems, government systems, transport vehicles, ordering systems, point-of-sales devices, and the like), and the like. In some cases, the sensor data may include LIDAR data, SIWIR data, red- green-blue image data, thermal data, Muon data, radio yvave data, yveight data, infrared data, and the like.

[0070] At 604, the simulation system may generate, based at least in part on the sensor data, a first 3D scene representing the facility over the period of time. For example, the system may generate a 3D immersive environment that may be consumed by a user via a 2D display and / or a 3D immersive display based on the sensor data associated with the facility over the period of time. In this manner, a user (e.g., such as a customer, potential customer, employee, third-party consultant, and the like) may view the operations of the facility (either independently or as a group within a shared scene) to evaluate the operations of the facility yvithout having to be physically present on-site. In some cases, the users consuming the content may be able to provide or attach comments (e.g., text, audio, visual data, or the like) to the first 3D scene so that the userand / or others may interact with the comments while they are consuming the first 3D scene at a later time.

[0071] At 606, the simulation system may receive object model data associated with one or more potential resources. For example, the simulation system may receive the object model data from third-party7systems, such as when the resource is equipment purchased from a vendor that allows for remote equipment testing or evaluation.

[0072] At 608, the simulation system may receive one or more user inputs introducing the one or more potential resources into the first 3D scene. For example, the user inputs may cause an existing resource to be swapped with the object model data associated with a potential new resource or the like.

[0073] At 610. the simulation system may generate, based at least in part on the sensor data, the object model data, the first 3D scene, and the user inputs, simulation data associated with the facility and, at 612, the simulation system may generate, based at least in part on the simulation data, one or more simulation instances associated with the faculty. For example, the simulation system may generate a simulation instance for each of the variety of different combinations that the user desires to evaluate within the facility.

[0074] At 614, the simulation system may generate, based at least in part an execution of individual ones of the one or more simulation instances, result data and a 3D scene. For example, the simulation system may execute each simulation instance to produce result data and a 3D scene for each individual simulation instance.

[0075] At 616, the simulation system may determine, based at least in part on the result data, one or more recommendations associated with the facility7. For example, the system may rank, score, or otherwise determine metrics associated with each option to modify the facility operations and determine a recommendation based on the rankings, scores, and metrics. For example, the system may determine a modification resulting in the largest increase in throughput for the facility7for a most common process and then recommend to a user that the facility implement the modification during physical operations.

[0076] At 618, the simulation system may augment the 3D scene with the result data and / or the one or more recommendations. For example, the system may determine changes or modifications to the operations of the facility7that may result in improvements to efficiency, safety, idle time, and the like based at least in part on thesensor data and / or historical data, best practices, and procedures, and / or user selected goals / initiatives.

[0077] At 620, the simulation system may display the 3D scene to a user. For example, a user may consume the 3D scene by interacting with the scene via a 3D immersive display system that allows the user to both view and interface (e.g., modify) the 3D scene as if the user was present on-site during the period of time.

[0078] FIG. 7 is an example simulation system 700 that may implement the techniques described herein according to some implementations. The system 700 may include one or more communication interface(s) 702 (also referred to as communication devices and / or modems), one or more processor(s) 704, and one or more computer readable media 706.

[0079] The system 700 can include one or more communication interfaces(s) 702 that enable communication between the system 700 and one or more other local or remote computing device(s) or remote sendees, such as a sensor system of FIG. 1. For instance, the communication interface(s) 702 can facilitate communication with other central processing systems, a sensor system, or other facility systems. The communications interfaces(s) 702 may enable Wi-Fi-based communication such as via frequencies defined by the IEEE 802.11 standards, short range wireless frequencies such as Bluetooth, cellular communication (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), satellite communication, dedicated short-range communications (DSRC), or any suitable wired or wireless communications protocol that enables the respective computing device to interface with the other computing device(s).

[0080] The system 700 may include one or more processors 704 and one or more computer-readable media 706. Each of the processors 704 may itself comprise one or more processors or processing cores. The computer-readable media 706 is illustrated as including memory / storage. The computer-readable media 706 may include volatile media (such as random access memory' (RAM)) and / or nonvolatile media (such as read only memory (ROM), Flash memory, optical disks, magnetic disks, and so forth). The computer-readable media 706 may include fixed media (e.g., RAM. ROM, a fixed hard drive, and so on) as well as removable media (e.g., Flash memory, a removable hard drive, an optical disc, and so forth). The computer-readable media 706 may be configured in a variety of other ways as further described below'.

[0081] Several modules such as instructions, data stores, and so forth may be stored within the computer-readable media 706 and configured to execute on the processors704. For example, as illustrated, the computer-readable media 706 stores data collection and processing instructions 708. 3D scene generation instructions 710, object model generation instructions 712, time period selection instructions 714, simulation instance generation instructions 716, simulation execution instruction 718, evaluation instructions 720, augmentation instructions 722, as well as other instructions, such as an operating system. The computer-readable media 706 may also be configured to store data, such as sensor data 724, process data 726, inventory data 728, resource data 730, simulation data 732, 3D scene data 734, machine learning models 736, training data 738, as well as other data.

[0082] The data collection and processing instructions 708 may be configured to receive and / or control the operations of various sensors throughout the facility in order to capture the sensor data 708 usable to generate 3D scenes representing the facility.

[0083] The 3D scene generation instructions 710 may be configured for each period of time indicated by the time period selection instructions 714, a 3D scene of the facilitybased at least in part on the sensor data 724 captured during the period of time.

[0084] The obj ect model generation instructions 712 may be configured to generate obj ect model data that may be used in the generation of 3D scenes, simulation data, simulation instances, and the like based at least in part on the resource data 730 associated with each or individual resources.

[0085] The time period selection instructions 714 may allow a user to select a period of time or interval of time for which the system 700 may generate a 3D scene from the sensor data.

[0086] The simulation instance generation instructions 716 may be configured to generate simulation data or instances using the available data (e.g., the sensor data 724, the process data 726, the inventory data 728, the resource data 730, the simulation data 732, the 3D scene data 734, and the like). In some cases, the simulation instance generation instructions 716 may utilize object models received from third parties such as potential new resources for the facility as discussed herein. The simulation instance generation instructions 716 may also utilize the one or more machine learning models 736 to execute the simulations, as discussed herein.

[0087] The simulation execution instruction 718 may execute the simulation instances and generate 3D scenes representing the execution. The simulation execution instruction 718 may also utilize the one or more machine learning models 736 to generate the report data or otherwise evaluate the result data, as discussed herein.

[0088] The evaluation instructions 720 may evaluate any result data generated by the simulation execution instruction 718 to generate report data, recommendations, and the like. The evaluation instructions 720 may also utilize the one or more machine learning models 736 to generate the simulation data or instances, as discussed herein.

[0089] The augmentation instructions 722 may be configured to overlay or augment the 3D scenes with facility data (e.g., safety data), such as the resource data (e.g., equipment and personnel data, assignments, shifts, statuses, and the like), third-party data 732 (e.g., orders, inventory assigned to orders, chain of custody data, and the like), process data 726 (e.g., process, methodologies, and the like associated wi th the facility), inventory data (e.g., inventory counts, location, age, expiration, expected consumption date, delivery date, and the like), and the like onto the 3D scene including associating the data with a corresponding resource or object within the 3D scene.

[0090] Although the discussion above sets forth example implementations of the described techniques, other architectures may be used to implement the described functionality and are intended to be within the scope of this disclosure. Furthermore, although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the claims.EXAMPLE CLAUSES

[0091] A.

[0092] While the example clauses described above are described with respect to one particular implementation, it should be understood that, in the context of this document, the content of the example clauses can also be implemented via a method, device, system, a computer-readable medium, and / or another implementation. Additionally, any of examples A-0 may be implemented alone or in combination with any other one or more of the examples A-O.CONCLUSION

[0093] While one or more examples of the techniques described herein have been described, various alterations, additions, permutations and equivalents thereof are included within the scope of the techniques described herein. As can be understood, the components discussed herein are described as divided for illustrative purposes. However, the operations performed by the various components can be combined or performed in any other component. It should also be understood that components or steps discussed with respect to one example or implementation may be used in conjunction with components or steps of other examples.

[0094] In the description of examples, reference is made to the accompanying drawings that form a part hereof, which show by way of illustration specific examples of the claimed subject matter. It is to be understood that other examples can be used and that changes or alterations, such as structural changes, can be made. Such examples, changes or alterations are not necessarily departures from the scope with respect to the intended claimed subject matter. While the steps herein may be presented in a certain order, in some cases the ordering may be changed so that certain inputs are provided at different times or in a different order without changing the function of the systems and methods described. The disclosed procedures could also be executed in different orders. Additionally, various computations that are herein need not be performed in the order disclosed, and other examples using alternative orderings of the computations could be readily implemented. In addition to being reordered, the computations could also be decomposed into sub-computations with the same results.

Claims

CLAIMS1. The method comprising: receiving first sensor data associated with a first physical environment over a first period of time; receiving object model data associated with one or more resource; generating, based at least in part on the first sensor data and the object model data, simulation data associated with the first physical environment; generating, based at least in part on the simulation data, at least two simulation instances; generating, based at least in part on an execution of a first simulation instance of the at least two simulation instances, a first three-dimensional scene; generating, based at least in part on an execution of a second simulation instance of the at least two simulation instances, a second three-dimensional scene; generating, based at least in part on the execution of the first simulation instance and the exaction of the second simulation instance, results data; augmenting the first three-dimensional scene with the results data; and displaying the first three-dimensional scene augmented with the results data to a user in a first immersive three-dimensional environment.

2. The method of claim 1, wherein the one or more resource includes one or more of facility equipment, personnel, or a group of resources.

3. The method of claims 1 or 2, further comprising receiving one or more user inputs modifying the one or more resources prior to generating the simulation data.

4. The method of any of claims 1-3, further comprising: augmenting the second three-dimensional scene with the results data; and displaying the second three-dimensional scene augmented with the results data to the user in a second immersive three-dimensional environment.

5. The method of any of claims 2-3, wherein the one or more user inputs includes additional resource model data introducing a potential new resource into the first physical environment.

6. The method of any of claims 1-5, further comprising: determining, based at least in part on the result data, one or more recommendations associated with the first physical environment; and augmenting the first three-dimensional scene with the recommendations.

7. The method of any of claims 1-6, wherein: generating simulation data associated with the first physical environment comprises inputting the first sensor data and the obj ect model data into one or more first machine learning models and receiving as an output of the one or more first machine learning models the simulation data, the one or more first machine learning models trained on object model data representing various resources and data captured over a multiple periods of time at multiple physical environments; generating the at least two simulation instances comprises inputting the simulation data into one or more second machine learning models and receiving as an output of the one or more second machine learning models the at least two simulation instances, the one or more second machine learning models trained on data captured over a multiple periods of time at multiple physical environments; and generating the first three-dimensional scene comprises inputting the output of the execution of the first simulation instance into one or more third machine learning models and receiving as an output of the one or more third machine learning models the first three-dimensional scene, the one or more first machine learning models trained on simulations associated with multiple periods of time at multiple physical environments.

8. The method of any of claims 1-7, further comprising generating report data based at least in part on the result data, the report data including at least one recommendation associated with improving operations of the first physical environment.

9. The method of any of claims 1-8, wherein the first physical environment is at least one of a shipping facility, a port of call, a logistics facility , a warehouse, a retail establishment, an assembly faci 1 i ty. a processing faci 1 i ty , a manufacturing faci 1 i ty , or a storage facility.

10. A computer program product comprising coded instructions that, when run on a computer, implement a method as claimed in any of claims 1-9.

11. A system comprising: a configuration system to generate, based at least in part on simulation data associated with a first physical environment, one or more simulation instances associated with the first physical environment; an execution system to receive the one or more simulation instances from the configuration system and to generate a three-dimensional scene associated with the first physical environment and individual instances of the one or more simulation instances and results data associated with the one or more simulation instances; and a user interface system configured to allow the user to consume the three- dimensional scenes in an immersive viewing environment.

12. The system of claim 11 , wherein the individual three-dimensional scenes are augmented with the result data to allow a user to consume the result data while engaged with the user interface system and the immersive viewing environment.

13. The system of claims 11 or 12, wherein the configuration system is configured to generate the one or more simulation instances based at least in part on sensor data associated with the first physical environment and object model data associated with resources associated with the first physical environment.

14. The system of any of claims 11-13, wherein the object model data is received from a third-party system, the third-part}7system unrelated to the first physical environment.

15. The system of any of claims 11-14, wherein the configuration system is configured to generate the one or more simulation instances based at least in part on process data associated with operations performed at the first physical environment.

Images