Systems and methods for supporting assembly of a collection of objects

Abstract

The present disclosure provides automated systems and methods for supporting assembly of a collection of objects used for a task. In certain embodiments, the systems and methods support assembly of surgical trays for use in a surgical or other medical setting. A method may comprise receiving images from a camera. The images may be analyzed to automatically identify objects in the images and to automatically determine that the objects have been placed in a container. A container inventory corresponding to a task for the container may be updated that the objects have been placed in the container. Machine vision modules and checking a weight of a container may both aid in determining whether the objects have been placed in the container. Thus, the method may automate the process of assembly of a collection of objects that may lead to a reduction of errors.

Description

Attorney Docket No. : 2018997-0010SYSTEMS AND METHODS FOR SUPPORTING ASSEMBLY OF A COLLECTION OF OBJECTSPRIORITY APPLICATION

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 760,878, filed on February 20, 2025, the disclosure of which is hereby incorporated by reference herein in its entirety.TECHNICAL FIELD

[0002] This disclosure relates generally to automated systems and methods for supporting assembly of a collection of objects, in some embodiments, using machine vision technologies.BACKGROUND

[0003] Many tasks require a collection of specific objects, such as tools and / or instruments, to perform. For example, different surgical procedures use different collections of surgical tools and / or instruments. In many cases, prior to performing a task, such a collection of objects is assembled in a container, such as a tray, to aid in performance of the task. Such is the case for surgical procedures, for example, where tray assembly is a major ongoing operation for hospitals. Assembly of such collections may involve assembling dozens or hundreds of objects, which takes significant time to assemble. In some cases, many objects can look and feel very similar to others, for example, in size and / or shape, such that it can be difficult to distinguish between them during assembly. There is a need therefore for systems and methods that can aid in the rapid assembly of collections of objects for tasks, especially collections that are complex.SUMMARY

[0004] The present disclosure provides automated systems and methods for supporting the assembly of a collection of objects (e.g., surgical instruments) used for a task (e.g., a surgical procedure). In certain embodiments, the systems and methods support assembly of surgical trays Page 1 of 5713305099vlAttorney Docket No. : 2018997-0010for use in a surgical or other medical setting. A method may comprise receiving images from a camera. The images may be analyzed to automatically identify objects discernible in the images and to automatically determine that the objects have been assembled, e.g., placed in a container. A container inventory corresponding to a task for the container may be updated to indicate that the objects have been placed in the container. Machine vision modules may aid in determining whether the objects have been placed in the container. A weight of a container may be used to verify that contents of the container are correct. The method may comprise providing a notification (e.g., an indication) when a wrong object is identified and / or has been placed in the container. Thus, the method may automate aspects of the process of assembly of a collection of objects.

[0005] The present disclosure includes the recognition that assembly of a large number of objects and associated laborious work is prone to errors during assembly. For example, a wrong object or a wrong quantity of objects may be assembled. Multiple variations of assembly -hundreds or even thousands of similar assemblies - are common for various industries. Such a large number of variations further increase a probability of assembly errors. Errors in assembly may translate into high costs due to associated delays and disturbances when performing tasks using assembled collections. Such disturbances may be particularly costly in certain applications such as, for example, surgeries, where increased operation time and / or use of incorrect surgical tools can lead to significant complications in patients. Systems and methods disclosed herein may lead to a reduction of errors during assembly and improved assembly speed, for example, due to verification of objects being assembled using machine vision and / or measured weight and / or real time user feedback regarding the state of assembly provided to a user.

[0006] The present disclosure also provides interactive graphical user interfaces (GUIs) and associated methods for assisting in assembling a collection of objects used for a task. The GUIs may comprise an inventory widget for displaying a container inventory and associated notifications (e.g., when objects have been placed in a container). The GUIs may comprise an image widget for displaying notification for identifying objects in the images. In this way, an automated, real-time, interactive assistance to a use in assembling a collection of objects may be provided.

[0007] In some aspects, the present disclosure is directed to a method for supporting assembly of a collection of objects (e.g., surgical instruments) to be used for a task (e.g., a Page 2 of 5713305099vlAttorney Docket No. : 2018997-0010surgical procedure). In certain embodiments, the method includes receiving, by a processor of a computing device, images [e.g., as a video stream (e.g., a real time video stream)] from (e.g., directly from) a camera. In certain embodiments, the method includes automatically identifying and / or tracking, by the processor, objects discernable (e.g., segmentable and, optionally, classifiable) in the images. In certain embodiments, the method includes automatically determining, by the processor, that the objects have been placed in a container (e.g., a tray). In certain embodiments, the method includes updating, by the processor, a container inventory for the container corresponding to a task to indicate that the objects have been placed in the container (e.g., are in the container).

[0008] In certain embodiments, the identifying and / or tracking objects and the determining that the objects have been placed in the container are each performed, by the processor, using the images. In certain embodiments, the container is discernable in the images. In certain embodiments, the determining that the objects have been placed in the container comprises determining that the objects have moved to a region. In certain embodiments, the region is discernible in the images. In certain embodiments, the region is outside of a field of view of the images. In certain embodiments, the determining that the objects have been placed in the container comprises determining, by the processor, that the objects are present in a region (e.g., area or volume) (e.g., with a computer vision module). In certain embodiments, the objects are individually determined to be present in the region. In certain embodiments, a method comprises automatically determining, by the processor, the region based on a position of the container in the images. In certain embodiments, determining the region comprises segmenting and classifying, by the processor, the container (e.g., with a computer vision module). In certain embodiments, the region is a first region and the method comprises, for each of the objects, automatically identifying, by the processor, that the object is present in a second region outside of the first region. In certain embodiments, the region is a first region and the method comprises, for each of the objects, automatically identifying, by the processor, the object when the object is present in a second region outside of the first region.

[0009] In certain embodiments, the determining that the objects have been placed in the container comprises tracking, by the processor, (e.g., with a machine vision module) the objects to determine that the objects have moved from the second region to the first region. In certain embodiments, the determining that the objects have been placed in the container comprises Page 3 of 5713305099vlAttorney Docket No. : 2018997-0010individually determining, by the processor, that each of the objects has been placed in the container. In certain embodiments, the updating the container inventory comprises updating, by the processor, the container inventory as each of the objects has been placed in the container. In certain embodiments, a method comprises determining, by the processor, from the images that one or more of the objects have been removed from the container. In certain embodiments, a method comprises updating, by the processor, the container inventory to indicate that the one or more of the objects is not in the container. In certain embodiments, a method comprises automatically determining, by the processor, that the container inventory is complete based on a determination that all of the objects are being disposed in the container at a same time. In certain embodiments, the method comprises automatically providing, by the processor, a notification (e.g., rendering and displaying a graphic or providing a sound) to a user in response to determining that the container inventory is complete. In certain embodiments, the method comprises providing, by the processor, a notification (e.g., rendering and displaying a graphic or providing a sound) to a user as each of the objects is determined to be placed in the container. In certain embodiments, the objects are individually determined to be placed in the container.

[0010] In certain embodiments, the container is discernible in the images. In certain embodiments, the camera comprises one or more cameras. In certain embodiments, the camera comprises sensitivity to at least a part of a spectrum of visible light (e.g., infrared, near-infrared, ultra-violet). In certain embodiments, the camera comprises an exposure time such that the objects are discernible in the images (e.g., not blurred). In certain embodiments, the camera comprises a focal length and / or a focal depth such that the objects remain in focus. In certain embodiments, a method comprises obtaining the images with the camera. In certain embodiments, the images have been obtained as a video stream (e.g., a real time video stream). In certain embodiments, a method comprises obtaining the images using the camera that is disposed to capture the first region and / or the second region. In certain embodiments, a method comprises showing (e.g., displaying, e.g., by a user, e.g., in the first and / or second region) (e.g., individually) each of the objects to the camera (e.g., individually) [e.g., in a region outside of (e.g., adjacent to) a container] (e.g., a second region). In certain embodiments, the showing comprises presenting each of the objects (e.g., individually) to the camera in a field of view of the camera. In certain embodiments, each of the objects is at least 80% (e.g., 50%, 60%, 70%, 90%, 95%) unobstructed during the presenting.Page 4 of 5713305099vlAttorney Docket No. : 2018997-0010

[0011] In certain embodiments, the identifying and / or tracking objects comprises segmenting and / or classifying, by the processor, (e.g., with a machine vision module) the objects in the images. In certain embodiments, the classifying the objects in the images comprises determining, for each of the objects, by the processor, whether the object is to be placed into the container according to the container inventory (e.g., whether the object identifies as an entry in the container inventory or not, whether a number of objects of a same kind as the object have been already placed in the container). In certain embodiments, the container inventory comprises entries that specify, for each of the objects, the object and a number of the object to be placed in the container. In certain embodiments, each entry in the list comprises a type of object (e.g., a name, an identification) and a quantity (e.g., a remaining quantity of objects of a same type to be placed in the container). In certain embodiments, the identifying and / or tracking objects discernable in the images comprises, for each of the objects, determining an entry in the list associated with the object (e.g., by classifying the object using the container inventory). In certain embodiments, the updating the container inventory comprises, for each of the objects, updating the quantity of the entry of the object.

[0012] In certain embodiments, the identifying and / or tracking comprises identifying the objects. In certain embodiments, the identifying is performed with a machine vision module. In certain embodiments, the identifying and / or tracking comprises tracking the objects. In certain embodiments, the tracking is performed with a machine vision module.

[0013] In certain embodiments, the determining that the objects have been placed in the container is performed with a machine vision module. In certain embodiments, the machine vision module comprises an image classification module. In certain embodiments, the machine vision module comprises a semantic segmentation module. In certain embodiments, the machine vision module comprises an instance segmentation module. In certain embodiments, the machine vision module comprises an image classification with localization module. In certain embodiments, the machine vision module comprises an object recognition module. In certain embodiments, the machine vision module comprises an object detection module. In certain embodiments, the machine vision module comprises a pattern recognition module. In certain embodiments, the machine vision module comprises an edge detection module. In certain embodiments, the machine vision module comprises a feature matching module. In certain embodiments, the machine vision module comprises a deep learning module.Page 5 of 5713305099vlAttorney Docket No. : 2018997-0010

[0014] In certain embodiments, the determining that the objects have been placed in the container comprises: receiving, by the processor, a weight for the container; and verifying, by the processor, contents of the container are correct based on the weight. In certain embodiments, a method comprises receiving, by the processor, a weight for the container. In certain embodiments, the determining that the objects have been placed in the container is based on the weight. In certain embodiments, a method comprises receiving, by the processor, a weight for the container; and determining, by the processor, that one or more of the objects have been placed in the container based on the weight. In certain embodiments, a method comprises receiving, by the processor, a weight for the container. In certain embodiments, a method comprises updating, by the processor, the container inventory based on the weight. In certain embodiments, a method comprises receiving, by the processor, a weight for the container. In certain embodiments, a method comprises determining, by the processor, that the weight is incorrect for one or more of the objects that have been placed in the container. In certain embodiments, a method comprises updating, by the processor, the container inventory (e.g., to remove an object) based on the weight. In certain embodiments, a method comprises preventing (e.g., stopping), by the processor, an update of the container inventory based on the weight [e.g., that would have otherwise been updated based on the images (e.g., based on identifying and / or tracking of objects discernible therein)]. In certain embodiments, a method comprises weighing the container with a balance to determine the weight. In certain embodiments, the container is disposed on the balance throughout a period of time during which the images were (e.g., or are) acquired.

[0015] In certain embodiments, the container comprises a tag (e.g., a QR code, barcode, RFID tag) (e.g., a printed tag) (e.g., a marking) (e.g., disposed on an upward facing surface of the container) and the method comprises selecting, by the processor, the container inventory based on the tag. In certain embodiments, selecting the container inventory comprises reading, by the processor, the tag using the images (e.g., using a machine vision module). In certain embodiments, a method comprise providing, by the processor, information corresponding to the tag. In certain embodiments, the container inventory is selected based on the information. In certain embodiments, a method comprises obtaining the information. In certain embodiments, obtaining the information comprises scanning the tag.Page 6 of 5713305099vlAttorney Docket No. : 2018997-0010

[0016] In certain embodiments, a method comprises providing, by the processor, a graphical user interface (GUI) (e.g., rendering and / or displaying the GUI on a screen) for user assistance (e.g., display the notification, receive user input, display the images). In certain embodiments, the GUI provides an automated, real-time, interactive assistance to the user in the assembling the collection of objects used for the task. In certain embodiments, the GUI comprises an inventory widget for displaying the container inventory. In certain embodiments, the updating the container inventory comprises updating the inventory widget and / or displaying one or more associated notifications in the inventory widget. In certain embodiments, the GUI comprises an image widget for displaying the images. In certain embodiments, the identifying and / or tracking objects comprises, for each of the objects, displaying a notification (e.g., as an overlay) in the image widget. In certain embodiments, the GUI comprises an indication widget for displaying indication messages to the user. In certain embodiments, the identifying and / or tracking objects comprises, for each of the objects, displaying the indication messages (e.g., a thumbnail, ID, and / or name associated with an object) in the indication widget. In certain embodiments, the GUI comprises a tracking status widget for displaying status messages to the user. In certain embodiments, the identifying and / or tracking objects and the determining that the objects have been placed in the container comprise, for each of the objects, displaying a status messages (e.g., “Tracking”, “Placed”, “Wrong Instrument”) in the tracking status widget.

[0017] In certain embodiments, a method is performed in real time. In certain embodiments, a method is performed in real time as the images are acquired. In certain embodiments, a method is performed in real time as a user assembles the objects in the container. In certain embodiments, a method is performed as the objects are assembled into the container (e.g., in real time). In certain embodiments, the objects are assembled in a sterile environment. In certain embodiments, upon determination that all of the objects are in the container, the objects are sterilized (e.g., in the container) (e.g., and packaged for sterile storage and / or use). In certain embodiments, the collection of objects comprises surgical instruments. In certain embodiments, a method comprises sterilizing the objects while in the container. In certain embodiments, the collection of objects comprises meal components (e.g., foods, beverages). In certain embodiments, the collection of objects comprises laboratory and research instruments and consumables. In certain embodiments, the collection of objects comprises components of medication packaging. In certain embodiments, the collection of objects comprises dentistry Page 7 of 5713305099vlAttorney Docket No. : 2018997-0010instruments and consumables. In certain embodiments, the collection of objects comprises tools and kits for vehicle manufacturing and repair workstations. In certain embodiments, the collection of objects comprises elements of medical kits and repair kits. In certain embodiments, the collection of objects comprises elements and consumable for circuit board assembly and electronic device repair kits. In certain embodiments, the collection of objects comprises instruments and consumables for beauty and skincare treatments and kits. In certain embodiments, the collection of objects comprises elements and consumable of emergency response kits and trauma trays (e.g., for ambulances).

[0018] In certain embodiments, the task comprises a surgical procedure. In certain embodiments, the task comprises food consumption. In certain embodiments, the task comprises an experimental (e.g., laboratory, research) procedure. In certain embodiments, the task comprises pharmaceutical manufacturing (e.g., packaging). In certain embodiments, the task comprises a dental procedure. In certain embodiments, the task comprises vehicle manufacturing and / or vehicle repair. In certain embodiments, the task comprises assembly and / or repair of electronic equipment (e.g., circuit boards). In certain embodiments, the task comprises a beauty and / or skincare treatment. In certain embodiments, the task comprises a medical emergencyresponse procedure.

[0019] In some aspects, the present disclosure is directed to a method for supporting assembly of a collection of objects (e.g., surgical instruments) to be used for a task (e.g., a surgical procedure). In certain embodiments, a method comprises receiving, by a processor of a computing device, a weight of a container and images from a camera in which one or more objects are discernable. In certain embodiments, a method comprises determining, by the processor, that the one or more objects have been placed in the container based on at least the images and the weight. In certain embodiments, determining that the one or more objects have been placed in the container based on the images comprises identifying and / or tracking the one or more objects using the images. In certain embodiments, the identifying and / or tracking comprises segmenting the one or more objects in the images. In certain embodiments, the identifying and / or tracking comprises classifying the one or more objects in the images. In certain embodiments, a method comprises updating, by the processor, a container inventory for the container corresponding to a task to indicate that the one or more objects have been placed in the container (e.g., are in the container).Page 8 of 5713305099vlAttorney Docket No. : 2018997-0010

[0020] In some aspects, the present disclosure is directed to a computer-implemented method for supporting assembly of a collection of objects to be used for a task. In certain embodiments, a method comprises, rendering and / or displaying on a screen, by the processor, one or more graphical user interface (GUIs), said one or more GUIs comprising an image widget for displaying a real time (e.g., live) stream of images (e.g., video stream). In certain embodiments, a method comprises receiving, by the processor, a determination that an object discernible in the images has been identified and / or tracked. In certain embodiments, a method comprises, in response to receiving the determination, providing (e.g., rendering and / or displaying) an indication, by the processor, in the one or more GUIs that the object belongs in a collection of objects used for a task.

[0021] In certain embodiments, a method comprises, for each of the objects, rendering and / or displaying a notification that indicates when the object has been identified and / or tracked. In certain embodiments, the notification is rendered and / or displayed in the image widget. In certain embodiments, the notification is rendered and / or displayed as an overlay over the object in the stream of images. In certain embodiments, a method comprises providing (e.g., rendering and / or displaying), by the processor, an indication in the one or more GUIs that the objects have been placed in the container (e.g., are in the container) [e.g., as the objects are placed in the container (e.g., individually)] (e.g., an individual respective indication).

[0022] In certain embodiments, the one or more GUIs comprise an inventory widget for displaying the container inventory and the providing an indication comprises, for each of the objects, rendering and / or displaying a notification to the user in the inventory widget. In certain embodiments, the one or more GUIs comprise a message widget for displaying a feedback message to the user and the method comprises displaying the feedback message (e.g., about system performance, frame rate, connectivity to the camera, guidance instructions) in the message widget. In certain embodiments, the one or more GUIs comprise an indication widget for displaying indication messages to the user and the method comprises, for each of the objects, displaying the indication messages (e.g., a thumbnail, ID, and / or name associated with an object) in the indication widget based on the determination received. In certain embodiments, the one or more GUIs comprise a tracking status widget for displaying status messages to the user and the method comprises, for each of the objects, displaying a status messages (e.g., “Tracking”, “Placed”, “Wrong Instrument”) in the tracking status widget.Page 9 of 5713305099vlAttorney Docket No. : 2018997-0010

[0023] In certain embodiments, a method comprises rendering and / or displaying on a screen, by the processor, a controls widget (e.g., a dialog box, drop down menu, radio button list) for entry of user display preferences (e.g., images, indications, notification preferences, organization of widgets in the one or more GUIs). In certain embodiments, a method comprises receiving, by the processor, a selection from the user display preferences. In certain embodiments, a method comprises rendering and / or display on a screen, by the processor, the one or more GUIs according to the selection from the user display preference.

[0024] In some aspects, the present disclosure is directed to a computer-implemented method for supporting assembly of a collection of objects to be used for a task via one or more graphical user interfaces (GUIs). In certain embodiments, a method comprises rendering and / or displaying on a screen, by the processor, one or more graphical user interface (GUIs), said one or more GUIs comprising an inventory widget for displaying the container inventory. In certain embodiments, a method comprises receiving, by the processor, a determination that an object discernible in the images has been identified and / or tracked. In certain embodiments, a method comprises, in response to receiving the determination, providing (e.g., rendering and / or displaying) an indication, by the processor, in the GUI that the object belongs in a collection of objects used for a task.

[0025] In certain embodiments, providing the indication comprises, for each of the objects, displaying a notification to the user in the inventory widget. In certain embodiments, the one or more GUIs comprise an image widget for displaying the images. In certain embodiments, the method comprises, for each object, rendering and / or displaying a notification that indicates when the object has been identified and / or tracked in the image widget.

[0026] In some aspects, the present disclosure is directed to a system for supporting assembly of a collection of objects to be used for a task. In certain embodiments, a system comprises a computing device comprising a processor and a memory having instructions stored thereon that, when executed by the processor, cause the processor to perform operations comprising a method presented herein. In certain embodiments, a system comprises a camera.

[0027] In some aspects, the present disclosure is directed to one or more non-transitory computer readable media having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform operations comprising a method presented herein.Page 10 of 5713305099vlAttorney Docket No. : 2018997-0010

[0028] In some aspects, the present disclosure is directed to a system for supporting assembly of a collection of objects to be used for a task. In certain embodiments, a camera for collecting images and a computing device for automatically identifying and / or tracking objects discernable in the images to determine when the objects are placed in a container. In certain embodiments, the computing device comprises a computer vision module that performs the identifying and / or tracking.

[0029] In certain embodiments, the camera comprises one or more cameras. In certain embodiments, the camera is an overhead camera. In certain embodiments, the camera is disposed such that a field of view of the camera comprises a workspace for assembling the collection of objects. In certain embodiments, the camera comprises sensitivity to at least a part of a spectrum of visible light (e.g., infrared, near-infrared, ultra-violet). In certain embodiments, the camera comprises an exposure time such that the objects are discernible in the images (e.g., not blurred). In certain embodiments, the camera comprises a focal length and / or a focal depth such that the objects remain in focus. In certain embodiments, the camera is disposed such that a field of view of the camera comprises a region (e.g., volume, area) associated with the container. In certain embodiments, the camera is disposed such that, for each of the objects, a field of view of the camera comprises at least 80% (e.g., 50%, 60%, 70%, 90%, 95%) of unobstructed view of the object (e.g., as measured by a projection area of the object in a particular position).

[0030] In certain embodiments, the system comprises a balance for weighing the container and the objects. In certain embodiments, the container is disposed on the balance throughout a period of time during which the images were (e.g., or are) acquired. In certain embodiments, for each of the objects, the identifying and / or tracking is based at least partly on a weight of the object determined by the balance. In certain embodiments, for each of the objects, the determination that the object is placed in the container is based at least partly on a weight of the object determined by the balance. In certain embodiments, the system comprises a container inventory. In certain embodiments, for each of the objects, the container inventory is updated based on a weight of the object determined by the balance.

[0031] Any two or more of the features described in this specification, including in this summary section, may be combined to form implementations of the disclosure, whether specifically expressly described as a separate combination in this specification or not.Page 11 of 5713305099vlAttorney Docket No. : 2018997-0010

[0032] At least part of the methods, systems, and techniques described in this specification may be controlled by executing, on one or more processing devices, instructions that are stored on one or more non-transitory machine-readable storage media. Examples of non-transitory machine-readable storage media include read-only memory, an optical disk drive, memory disk drive, and random access memory. At least part of the methods, systems, and techniques described in this specification may be controlled using a computing system comprised of one or more processing devices and memory storing instructions that are executable by the one or more processing devices to perform various control operations.DEFINITIONS

[0033] In order for the present disclosure to be more readily understood, certain terms used herein are defined below. Additional definitions for the following terms and other terms may be set forth throughout the specification.

[0034] In this application, unless otherwise clear from context or otherwise explicitly stated, (i) the term “a” may be understood to mean “at least one”; (ii) the term “or” may be understood to mean “and / or”; (iii) the terms “comprising” and “including” may be understood to encompass itemized components or steps whether presented by themselves or together with one or more additional components or steps; (iv) the terms “about” and “approximately” may be understood to permit standard variation as would be understood by those of ordinary skill in the relevant art; and (v) where ranges are provided, endpoints are included. In certain embodiments, the term “approximately” or “about” refers to a range of values that fall within 25%, 20%, 19%, 18%, 17%, 16%, 15%, 14%, 13%, 12%, 11%, 10%, 9%, 8%, 7%, 6%, 5%, 4%, 3%, 2%, 1%, or less in either direction (greater than or less than) of the stated reference value unless otherwise stated or otherwise evident from the context (except where such number would exceed 100% of a possible value).BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present teachings described herein will be more fully understood from the following description of various illustrative embodiments, when read together with the accompanying drawings. It should be understood that the drawing described below is for illustration purposes only and is not intended to limit the scope of the present teachings in any Page 12 of 5713305099vlAttorney Docket No. : 2018997-0010way. The foregoing and other objects, aspects, features, and advantages of the disclosure will become more apparent and may be better understood by referring to the following description taken in conjunction with the accompanying drawings.

[0036] FIG. 1 is a view of a tray object definition checklist with guiding pictures, according to illustrative embodiments of the present disclosure.

[0037] FIG. 2 is a view of a technician assembling a tray with instruments at an assembly station, according to illustrative embodiments of the present disclosure.

[0038] FIGs. 3A-3B are views of a system for assisting a user to assemble a collection of objects used for a task, according to illustrative embodiments of the present disclosure.

[0039] FIG. 3C is a schematic diagram illustrating components in a system for assembling a collection of objects shown in FIGs. 3A-3B, according to illustrative embodiments of the present disclosure.

[0040] FIG. 4 is a flow diagram showing a method of assembling a collection of objects, according to illustrative embodiments of the present disclosure.

[0041] FIGs. 5A-5G are a series of views of a graphical user interface for assisting a user to assemble a collection of objects used for a task, according to illustrative embodiments of the present disclosure.

[0042] FIGs. 5H-5L are a series of views of a graphical user interface for verifying whether the contents of the container are correct, according to illustrative embodiments of the present disclosure.

[0043] FIG. 6 is a flow diagram showing a method of assisting a user to assemble a collection of objects via one or more graphical user interfaces (GUIs), according to illustrative embodiments of the present disclosure.

[0044] FIG. 7 is a schematic diagram illustrating components in a system for assembling a collection of objects via one or more graphical user interfaces (GUIs), according to illustrative embodiments of the present disclosure.

[0045] FIG. 8 is a block diagram of an example network environment for use in the methods and systems described herein, according to illustrative embodiments of the present disclosure.

[0046] FIG. 9 is a block diagram of an example computing device and an example mobile computing device, for use in illustrative embodiments of the present disclosure.Page 13 of 5713305099vlAttorney Docket No. : 2018997-0010DETAILED DESCRIPTION OF CERTAIN EMBODIMENTS

[0047] Assembling a collection of objects may be a tedious task. For example, the assembly of surgical instruments into a tray for a surgical procedure involves hundreds of instruments. A layout and contents of a tray are usually defined as a checklist either on a paper or as a digitized version. FIG. 1 shows an example of a tray layout and individual instruments that a technician follows by visual inspection. Several instruments have a very similar shape and size. The checklists are difficult to follow and a technician using them may be confused in case of similar instruments. In clinical settings, hundreds or even thousands of tray definitions and various instruments are common. FIG. 2 shows a process of a surgical tray assembly. Multiple forceps-like instruments of various sizes are shown in FIG.2, demonstrating complexity of the assembly process.

[0048] In a typical process, technicians work in a sterile processing department assembling trays. A technician refers to a checklist to place relevant instruments in a tray. The instruments may come directly from a cleaning and washing process or from a storage. Once a tray is complete, it is closed, a special wrapping is applied, and the tray is sent for sterilization. After sterilization, the tray is placed into a sterile storage, where it remains until a surgical procedure.

[0049] There are several challenges with the described process. The list of tray items is rarely dynamic and requires manual adjustments if requirements of an underlying process (e.g., surgical procedure) change. For a technician, it may be difficult to avoid mixing different instruments, keep track of progress, follow changes in the tray composition, and identify all potential mistakes. The tedious process of checking the list, locating a correct instrument, placing it in the tray, and repeating the steps takes a significant amount of time. As a result, errors (incorrect instruments in trays) or omissions (missing instruments) are common.

[0050] The impact of these errors is substantial because the contents of a tray often may not be verified without opening it, whereas the opening compromises its sterility. Consequently, errors are often discovered only during a surgical procedure, causing significant delays as the missing or incorrect instruments must be located and delivered. Thus, the errors lead to unnecessarily increased procedure costs because of added time and overhead. Moreover, patientPage 14 of 5713305099vlAttorney Docket No. : 2018997-0010outcomes may be negatively affected as longer procedure durations are associated with higher infection rates.1. System and Elements Thereof

[0051] Systems of the present disclosure may rely on a camera (e.g., one or more cameras) to collect images of objects. Assembly of objects may be subsequently analysed (e.g., to identify objects, determine their state, track their trajectories) to obtain associated information. Information obtained from collected images may be further used to automate assembly of objects. For example, the information may be used to guide assembly of objects (e.g., by providing notifications if an error has been made, by providing directions about an assembly order). In certain embodiments, system may rely on images (i.e., without access to a camera). Images may comprise videos, video stream, stream of images. Images may be transmitted in real-time (e.g., without a significant delay to a user). Images may be on-demand (e.g., started, paused, resumed, and stopped by a user).

[0052] FIGs. 3A-3B show different views and FIG. 3C shows a schematic of the system 300 for assembling a collection of objects used for a task. The system 300 comprises the camera 301, the computing and / or displaying device 302, the container 303, the balance 304, the collection of objects to be assembled 305, and the working surface 306. An assembly of objects may be performed at least partially in the field of view of the camera 301. The camera 301 may collect images. The camera 301 may be connected to the computing and / or displaying device 302. The computing and / or displaying device 302 may receive images from the camera 301, identify objects out of the collection of objects 305 discernible in the images, and determine that the objects have been placed in the container 303. The container 303 may be positioned on the balance 304. The balance 304 may be connected to the computing and / or displaying device 302, providing real-time weight measurements (e.g., of the container 303 and its content). In certain embodiments, a task may be associated with a container inventory - a list of entries corresponding to specific objects and their quantity to be placed into a container. The computing and / or displaying device 302 may update a container inventory to indicate that objects have been placed in the container.

[0053] A camera may correspond to an optical system for capturing images. A camera may correspond to one or more cameras. A camera may be characterized by a field of view - an Page 15 of 5713305099vlAttorney Docket No. : 2018997-0010area or volume that is captured by the camera in a form of images. A camera may be positioned so that assembly of objects is at least partially performed in a field of view of the camera. A camera may be sensitive to at least a part of a spectrum of visible light (e.g., near-infrared, infrared, ultra-violet). A camera may have a specific focus length and / or focal depth so that a working surface and objects remain in focus (e.g., throughout the assembly process). A camera may be positioned so that a working surface and objects remain in focus (e.g., throughout the assembly process). A camera may capture images of objects with sufficient details (e.g., to identify objects). Image size may be not too large to make a processing time of the image sufficiently faster (e.g., smaller than a camera exposure time, e.g., not to create a backlog of image processing in, e.g., real-time operation). A processing time of an image may be sufficiently fast (e.g., smaller than a camera exposure time, e.g., not to create a backlog of image processing in, e.g., real-time operation). A connectivity of a camera to a computing and / or displaying device may be set up so that images captured by the camera are transferred to the computing and / or displaying device faster than a camera exposure time (i.e., no backlog of images is created in, e.g., real-time operation).

[0054] Example of camera parameters: (1) system field of view (e.g., a working station): 2x1 m2; (2) working distance (e.g., between a camera and a working station): 2 m; (3) maximal moving object speed: 1 m / s; (4) minimal object size: 50 mm; (5) minimal object image size: 75 px; and (6) maximal acceptable blur value: 1 mm. In certain embodiments, a field of view of a camera is approximately 2x1 m2(e.g., 4x1 m2, 0.5x1 m2, 0.3x1 m2, 2x2 m2, 2x3 m2, 2x0.5 m2, 2x0.3 m2, 1x1 m2, 0.5x0.5 m2, 0.3x0.3 m2, 0.2x0.2 m2, 0.1x0.1 m2). In certain embodiments, a working distance between a camera and a working surface is approximately 2 m (e.g., 5 m, 3 m, 1 m, 0.5 m, 0.3 m). In certain embodiments, maximal moving object speed (e.g., within a field of a camera) is approximately 1 m / s (e.g., 2 m / s, 5 m / s, 10 m / s, 20 m / s). In certain embodiments, minimal object size is approximately 50 mm (e.g., 100 mm, 30 mm, 20 mm, 10 mm, 5 mm, 1 mm). In certain embodiments, minimal object image size is approximately 75 px (e.g., 100 px, 50 px ,25 px, 10 px).

[0055] Example of camera minimal characteristics: (1) exposure time: 1000 ps; (2) frames-per-second (FPS): 30 frames / s; (3) spatial resolution: 1.5 px / mm; (4) minimal camera sensor resolution: 3000 px; (5) sensor resolution: 8.3 megapx; and (6) data rate: 2.1 gigabit. In certain embodiments, minimal exposure time of a camera is approximately 1 ms (e.g., 10 ms, 20Page 16 of 5713305099vlAttorney Docket No. : 2018997-0010ms, 0.5 ms, 0.2 ms, 0.1 ms). In certain embodiments, frame-per-second (FPS) of a camera is approximately 30 frames / s (e.g., 20 frames / s, 10 frames / s, 5 frames / s, 2 frames / s, 1 frame / s, 60 frames / s, 120 frames / s). In certain embodiments, spatial resolution of a camera is approximately 1.5 px / mm (e.g., 2.0 px / mm, 5.0 px / mm, 10.0 px / mm, 25 px / mm, 50 px / mm, 100 px / mm, 1.0 px / mm, 0.5 px / mm). In certain embodiments, minimal camera sensor resolution is approximately 3000 px (e.g., 5000 px, 10000 px, 20000 px, 2000 px, 1000 px). In certain embodiments, sensor resolution is approximately 8.3 megapx (e.g., 10 megapx, 15 megapx, 20 megapx, 30 megapx, 50 megapx, 5 megapx, 2 megapx, 1 megapx). In certain embodiments, data rate from a camera and / or to a computing devise is approximately 2.1 gigabit (e.g., 3 gigabit, 5 gigabit, 10 gigabit, 1 gigabit, 0.5 gigabit, 0.1 gigabit).

[0056] In certain embodiments, a camera comprises one or more cameras. In certain embodiments, a balance comprises additional sensors (e.g., thermal imaging, X-rays, radio frequency identification (RFID) readout, electromagnetic and conductivity sensors).

[0057] A computing and / or displaying device may identify objects discernible in images for their subsequent tracking. Each entry in a container inventory may be associated with specific features to be identified in images. A machine vision module may be used for object identification. Images may be segmented and / or classified to identify an object. Specific features may comprise a reference set of images of an object. Specific features may comprise an individual module (e.g., a trained deep learning network) to identify individual object (e.g., a specific type or kind of objects).

[0058] A user may be notified about a confidence of object identification. A user may be notified if a confidence of object identification is below 50 % (e.g., 60 %, 70 %, 80%, 90 %, 95 %, 99 %). A confidence of object identification may be based on a statistical measure of object properties belonging to specific features associated with an entry in a container inventory. A confidence of object identification may be displayed to a user in real-time so that the user may display the object (e.g., from different perspectives, unobstructed) to a camera in a field of view of the camera in attempt to improve the confidence.

[0059] A computing and / or displaying device may determine whether identified objects have been placed in a container by, for example, tracking each of the objects into a region associated with a container (e.g., area, volume). The region may be defined by a user or may be (e.g., automatically) determined (e.g., by a computing and / or displaying device). The region Page 17 of 5713305099vlAttorney Docket No. : 2018997-0010may be outside of a field of view of a camera. For example, a region may be next to a specific boundary of a field of view of a camera and tracking an object across the boundary may be associated with placing the object into a container. An object has been placed in a container if, for example, the object remains in the region for a specific time. An object has been placed in a container if, for example, a speed of the object (e.g., as determined from images) falls below a certain threshold (e.g., while the object is in a region). An object has been placed in a container if, for example, the object remains in a region even after an assistive element (e.g., a hand of an operator, a robotic arm), by which it has been placed into the region, is elsewhere.

[0060] A region associated with a container may be a first region. A region (e.g., area, volume) that is associated with being outside of a container may be defined as a second region. For example, a region associated with a working area outside of a container may be a second region. The second region may be in a field of view of a camera (e.g., or in an image). The second region may be outside of a field of view of a camera. For example, only a first region is in a field of view of a container and an object is tracked as the object appears in the field of view of the camera. The ability to identify whether an object has been placed in a container may comprise tracking that the object from the second region into the first region (e.g., and not back).

[0061] A computing and / or displaying device may display a container inventory. Once a container inventory is updated, a computing and / or displaying device may display an updated container inventory. An update to a container inventory may be associated with updating a quantity of remaining objects (e.g., of a certain kind or type). An update to a container inventory may be associated with displaying information associated with only remaining objects.

[0062] FIG. 4 shows a flow diagram of a method for assembling a collection of objects used for a task. The method 400 comprises receiving images from a camera at step 401. At step 402, identifying and / or tracking objects discernible in the images. At step 403, determining that the objects have been placed in a container. At step 404, updating a container inventory to indicate that the objects have been placed in the container.

[0063] In certain embodiments, determining that objects have been placed in a container comprises determining that the objects are present in a region (e.g., area or volume) (e.g., with a computer vision module). The objects may be individually determined to be present in a region.Page 18 of 5713305099vlAttorney Docket No. : 2018997-0010

[0064] In certain embodiments, a method comprises automatically determining a region based on a position of a container in images. The determining the region may comprise segmenting and classifying the container (e.g., with a computer vision module).

[0065] In certain embodiments, a region is a first region. A method may comprise, for each of objects, automatically identifying that the object is present in a second region outside of a first region.

[0066] In certain embodiments, determining that objects have been placed in a container comprises tracking (e.g., with a machine vision module) the objects to determine that the objects have moved from a second region to a first region. The determining that the objects have been placed in the container may comprise individually determining that each of the objects has been placed in the container. The updating the container inventory may comprise updating the container inventory as each of the objects has been placed in the container.

[0067] In certain embodiments, a method comprises determining from images that one or more of objects have been removed from a container and updating a container inventory to indicate that the one or more of the objects is not in the container. In certain embodiments, a method comprises automatically determining that a container inventory is complete based on a determination that all of objects being disposed in the container at a same time. In certain embodiments, a method comprises automatically providing a notification (e.g., rendering and displaying a graphic or providing a sound) to a user in response to determining that a container inventory is complete.

[0068] In certain embodiments, a method comprising providing a notification to a user as each of objects is determined to be placed in a container. The notification may comprise rendering and displaying a graphic element. The notification may comprise a sound. The notification may comprise the following notifications. A first notification may be associated with an object determined to be placed in the container. A second notification may be associated with all the objects determined to be placed in the container (e.g., according to a container inventory). A third notification may be associated with a wrong object (e.g., according to a container inventory) determined to be placed in the container. A fourth notification may be associated with a wrong object determined to be removed from the container.

[0069] In certain embodiments, images have been obtained by a camera disposed to capture a first region. In certain embodiments, images have been obtained by a camera disposed Page 19 of 5713305099vlAttorney Docket No. : 2018997-0010to capture a second region. In certain embodiments, images have been obtained by a camera disposed to capture a first region and / or a second region.

[0070] In certain embodiments, a method comprises obtaining images using a camera that is disposed to capture a first region. In certain embodiments, a method comprises obtaining images using a camera that is disposed to capture a second region. In certain embodiments, a method comprises obtaining images using a camera that is disposed to capture a first region and / or second region.

[0071] In certain embodiments, identifying and / or tracking objects discernable in the images comprises showing (e.g., displaying, e.g., by a user, e.g., in a first and / or second region) at least some of the objects to a camera (e.g., in the camera field of view).Balance

[0072] A balance positioned under a container may provide an orthogonal measure for object identification and / or tracking. A balance may perform weight measurements of an object. Once an object is identified in an image, a balance may be reset to measure the weight of the object once the object is placed in a container. Once a previous object has been placed in a container, a balance may be reset to measure the weight of a next object. Once an object has been placed in a container (e.g., as determined using information from images), a weight of an object may be compared to an entry weight from a container inventory. If a weight difference between the weights is less than 2% (e.g., 0.5%, 1%, 5%, 10%), an object may be determined as has been placed in a container. If there is a weight difference between the weights more than 2% (e.g., 0.5%, 1%, 5%, 10%), a notification may be provided. The weight difference may be associated with identification of a wrong object. The weight difference may be associated with a defect in the object. For example, if confidence of object identification is high but there is a weight difference, a notification related to a defect of an object may be displayed (i.e., weight measurements detect defects that are not captured by object identification). For example, if confidence of object identification is low and there is a weight difference, a notification related to a defect of an object may be displayed (i.e., the defect distorts the objects so that the object can’t be identified). An object may be identified based on its weight (e.g., when identification via images is not available). An object may be identified based on a combination of its weight and its identification from images (e.g., despite a low value of a confidence of object identification).Page 20 of 5713305099vlAttorney Docket No. : 2018997-0010

[0073] In certain embodiments, determining that objects have been placed in a container comprises receiving a weight for the container and verifying contents of the container are correct based on the weight. The objects may be individually determined to be present in the container.

[0074] In certain embodiments, determining that objects have been placed in a container comprises weighing the container with a balance to determine the weight. The container may be disposed on the balance throughout a period of time during which images are or were acquired.Real-time Performance

[0075] In certain embodiments, a method is performed in real time. The method may be performed in real time as images are acquired, as the objects are assembled into a container. The ability to perform methods of the present disclosure in real time may provide its benefits. For example, an ability to correct user behavior may be needed if an error of placing a wrong instrument has been detected. For example, an ability to provide real time feedback to a user during performance may be needed if an object not from a container list is identified. For example, an ability to guide a user in assembly of objects may be needed if a user is unfamiliar with the assembly.2. Assembly Process and Graphical User Interfaces

[0076] An example of a method for assembling a collection of objects used for a task, according to the present disclosure, follows. With an empty container, a container inventory (e.g., instrument list) is loaded and initialized (e.g., displayed by an instrument widget). The inventory may be parametrized (e.g., manually, e.g., from a previous list) or loaded from a file. A container may have a special tag (e.g., quick response (QR) code, barcode, RFID), shape, colors, and / or weight to enable automatic load of associated container inventory. For example, a container may have a serial number. A database may provide mapping between the serial number and an associate container inventory. Objects may be assembled either in an arbitrary order or in an order defined by a container inventory. Images may be received from a camera. A notification may be provided once an object is identified and / or tracked (e.g., via a machine vision module). Tracking for objects discernible in images may be used to ensure an object has been placed in a container. A weight of an object (e.g., as measured after object placement into a container, e.g., as a difference between before and after object placement) may be used to determine whether the object has been placed in a container. A notification may be provided if Page 21 of 5713305099vlAttorney Docket No. : 2018997-0010there is a discrepancy between, for each of object, object placement in a container via object identification and / or tracking and object placement in the container via weight determination. The discrepancy may be used to identify an object defect (e.g., missing elements in the object, failure of the object). A notification may be provided once all objects are placed in a container according to a container inventory.

[0077] In certain embodiments, methods and systems presented herein comprise a graphical user interface (GUI) for assistance. The GUI may provide an automated, real-time, interactive assistance to a user in assembling a collection of objects used for a task. FIGs. 5A-5L show an example of such a GUI.

[0078] In certain embodiments, the GUI 500 comprises an inventory widget 501 for displaying a container inventory. The inventory widget 501 may comprise a list of all object entries (e.g., their names, IDs) to be assembled (e.g., in a container) and their associated quantities. The inventory widget 501 may comprise a remaining quantity for each of objects left to be assembled (e.g., in a container). The inventory widget 501 may comprise, for each of objects, object weight (e.g., with margins) and / or an object thumbnail image. The thumbnails may be shown in proximity to object entries. The inventory widget 501 may comprise tracking status for each of objects. The inventory widget may comprise, for each of objects, displaying a notification that the object has been assembled (e.g., placed into a container, is in a container), e.g., as shown by 501a in FIGs. 5D-5F, by 501b in FIG. 5F, and by 501z in FIG. 5G.

[0079] In certain embodiments, the GUI 500 comprises a control widget 502 for entry of user display preferences (e.g., type and color of images and indications, notification preferences, organization of widgets in the GUI). The control widget 502 may comprise a dialog box, a drop down menu, and / or a radio button list. The GUI 500 may be rendered and / or displayed on a screen according to selection from user display preferences. The user display preferences may comprise an option for displaying a first region (e.g., associated with an assembly area, a container) and / or a second region (e.g., outside of an assembly area, a container; e.g., on a working surface). The user display preferences may comprise an option to start and / or stop a live stream of images.

[0080] In certain embodiments, the GUI 500 comprises a message widget 503 for displaying feedback messages. For example, the feedback messages may comprise informationPage 22 of 5713305099vlAttorney Docket No. : 2018997-0010associated with system performance, frame rate, connectivity to a camera, and / or guidance instructions.

[0081] In certain embodiments, the GUI 500 comprises an image widget 504 for displaying a real time (e.g., live) stream of images (e.g., a video). The image widget 504 may comprise, for each of objects, displaying a notification (e.g., an overlay, a bounding box over area associated with the object) once the object has been identified (e.g., using a machine vision module), e.g., 504a. The image widget 504 may comprise determining and / or displaying a first region 508 (e.g., associated with an assembly area, a container) and a second region 509 (e.g., outside of an assembly area, a container, e.g., on a working surface).

[0082] In certain embodiments, the GUI 500 comprises an indication widget 505 for displaying indication messages. The indication widget 505 may comprise, for each of object, displaying an indication message (e.g., a thumbnail, ID, and / or name associated with the object), e.g., 505a.

[0083] In certain embodiments, the GUI 500 comprises a tracking status widget 506 for displaying status messages. The tracking status widget may comprise messages, such as “Waiting” (e.g., for an object associated with a container inventory), “Tracking” (e.g., an object is identified, e.g., via a machine vision module), “Wrong instrument” (e.g., an object not in a container inventory).

[0084] In certain embodiments, a method 600 for assembling a collection of objects used for a task comprises the following steps as shown in FIG. 6. At step 601, rendering and / or displaying a real time stream of images. At step 602, identifying and / or tracking objects discernible in the images. At 603, determining that the objects have been placed in a container. At step 604, providing an indication that the objects have been placed in the container.

[0085] In certain embodiments, methods of the present disclosure are performed by a computing device 700 as shown in FIG. 7. The device 700 comprises a memory 701, a process 702, a user interface module 703, and a machine vision module 704.3. Machine Vision Modules

[0086] In certain embodiments, identifying and / or tracking objects comprises segmenting and / or classifying (e.g., with a machine vision module) the objects in images. In certain embodiments, classifying objects in images comprises determining, for each of the objects, Page 23 of 5713305099vlAttorney Docket No. : 2018997-0010whether the object is to be placed into a container according to a container inventory (e.g., whether the object identifies as an entry in the container inventory or not, whether a number of objects of a same kind as the object have been already placed in the container). The container inventory may comprise a list of entries to specify a number and type of objects to be placed in the container. Each entry in the list may comprise a type of object (e.g., a name, an identification) and a quantity (e.g., a remaining quantity of objects of a same type to be placed in the container). The identifying and / or tracking objects may comprise, for each of the objects, determining an entry in the list associated with the object. Updating the container inventory may comprise, for each of the objects, updating the quantity of the entry of the object.

[0087] In certain embodiments, identifying objects is performed with a machine vision module. In certain embodiments, tracking objects is performed with a machine vision module. In certain embodiments, determining that objects have been placed in a container is performed with a machine vision module.

[0088] In certain embodiments, a machine vision module comprises an image classification module. In certain embodiments, a machine vision module comprises a semantic segmentation module. In certain embodiments, a machine vision module comprises an instance segmentation module. In certain embodiments, a machine vision module comprises an image classification with localization module. In certain embodiments, a machine vision module comprises an object recognition module. In certain embodiments, a machine vision module comprises an object detection module. In certain embodiments, a machine vision module comprises a pattern recognition module. In certain embodiments, a machine vision module comprises an edge detection module. In certain embodiments, a machine vision module comprises a feature matching module. In certain embodiments, a machine vision module comprises a deep learning module.

[0089] Various machine vision modules may be used with systems and methods presented herein. For example, algorithms described in U.S. Provisional Application No.63 / 547,112 “Systems and methods for operating room management” and No. 63 / 706,602 “Systems and methods for generating image datasets for training machine vision models,” the disclosures of each of which is incorporated by reference herein in its entirety, may be used. The objects may be introduced into a system using a specially conceived device, as described inPage 24 of 5713305099vlAttorney Docket No. : 2018997-001063 / 706,602 “Systems and methods for generating image datasets for training machine vision models,” the disclosure of which is incorporated by reference herein in its entirety.

[0090] Other object detection and tracking methods available commercially and / or in the literature may be use with systems and methods presented herein. For example, YOLO (You Only Look Once): a real-time object detection method using convolutional neural networks, where objects are inputted via bounding boxes or annotation tools. SSD (Single Shot MultiBox Detector): an efficient object detection approach that combines feature extraction and multi-scale predictions; where objects are inputted via bounding boxes or labeled datasets. Faster R-CNN (Region-Based Convolutional Neural Networks): a two-stage object detection method that identifies regions of interest and classifies them, where objects are inputted via bounding boxes or manual tagging. DeepSORT (Simple Online and Realtime Tracking with Deep Association Metric): a tracking algorithm that associates detected objects across frames and uses bounding boxes as input from detection methods like YOLO or Faster R-CNN. TrackNet: a deep learningbased tracking method designed for sports and other applications, focusing on motion prediction, where objects are inputted through bounding boxes or trajectory labels. SORT (Simple Online Realtime Tracker): a lightweight tracking algorithm that associates detected objects across frames using Kalman filters and uses bounding boxes from detection methods as input.RetinaNet: a detection method combining a feature pyramid network with focal loss to handle class imbalance, requiring bounding boxes and annotations for training. Mask R-CNN extends Faster R-CNN for instance segmentation, detecting objects and their pixel-level masks, and uses bounding boxes or segmentation masks as input. CenterNet: an object detection approach using keypoint estimation to locate object centers, where objects are inputted via labeled datasets or bounding box annotations. ByteTrack: an advanced multi-object tracker for object association across frames that uses bounding boxes from any detection model as input. Detectron2:Facebook Al’s modular object detection framework supporting various detection methods, where objects are inputted via labeled datasets, bounding boxes, or segmentation masks. OpenCV Multi-Object Tracking: a library-based solution that provides simple tracking algorithms like KCF and CSRT and uses bounding boxes or manual tagging as input. TLD (Tracking-Learning-Detection) combines tracking and detection for long-term object tracking, where objects are inputted through manual tagging or bounding boxes. FairMOT (Fair Multi-Object Tracking) simultaneously handles detection and re-identification using a unified framework and uses Page 25 of 5713305099vlAttorney Docket No. : 2018997-0010bounding boxes and re-ID annotations for input. Objects are typically inputted to these methods via manual tagging, pre-labeled datasets, bounding boxes, scanners, or automatic annotation tools, depending on the application and training requirements.4. Applications

[0091] Systems and methods disclosed herein enable automation of assembly of a collection of objects. The ability to perform such automation is particularly important in fields where there are multiple objects to be assembled and / or where there are multiple ways of how to assemble objects. As a general approach, systems and methods disclosed herein can be used whenever the ability to precisely monitor objects reduces associated errors.

[0092] In certain embodiments, a collection of objects comprises surgical instruments. In certain embodiments, a task comprises a surgical procedure. The surgical instruments may be handled (e.g., for assembly) in a sterile environment. The surgical instruments may be sterilized after the assembly and further stored in a sterile storage. In certain embodiments, a content of a tray containing surgical instruments can’t be verified without opening it and, as result, compromising its sterility. The importance of the assembly process in this case is associated with ensuring the presence of correct instruments and materials in the operating room.

[0093] In certain embodiments, a collection of objects comprises meal components (e.g., foods, beverages). In certain embodiments, a task comprises food consumption, e.g., preparation of meal trays for airlines, schools, and institutional catering. The importance of the assembly process in this case is associated with ensuring standardized meal presentation, portion control, and adherence to dietary requirements, such as allergens or special preferences.

[0094] In certain embodiments, a collection of objects comprises laboratory and research instruments and consumables. In certain embodiments, a task comprises an experimental (e.g., laboratory, research) procedure, e.g., preparation of instrument trays for experiments or chemical analysis. The importance of the assembly process in this case is associated with preventing contamination, ensuring proper sequencing of experimental steps, and reducing time spent searching for tools or materials.

[0095] In certain embodiments, a collection of objects comprises components of medication packaging. In certain embodiments, a task comprises pharmaceutical manufacturing (e.g., packaging), e.g., assembly of trays for medication packaging or quality control processes.Page 26 of 5713305099vlAttorney Docket No. : 2018997-0010The importance of the assembly process in this case is associated with guaranteeing consistent handling of sensitive materials, preventing cross-contamination, and adhering to regulatory standards.

[0096] In certain embodiments, a collection of objects comprises dentistry instruments and consumables. In certain embodiments, a task comprises a dental procedure, e.g., preparation of dental procedure trays with instruments for cleanings, fillings, or surgeries. The importance of the assembly process in this case is associated with ensuring proper sterilization and sequence of instruments, optimizing patient care and reducing procedural delays.

[0097] In certain embodiments, a collection of objects comprises tools and kits for vehicle manufacturing and repair workstations. In certain embodiments, a task comprises vehicle manufacturing and / or vehicle repair, e.g., assembly of toolkits for vehicle manufacturing or repair workstations. The importance of the assembly process in this case is associated with improving efficiency in production lines or repair bays by ensuring the availability of necessary tools in the correct order.

[0098] In certain embodiments, a collection of objects comprises elements of medical kits and repair kits (e.g., assembly of field medical kits or repair toolkits for equipment). The importance of the assembly process in this case is associated with ensuring that personnel can respond quickly in critical situations with properly organized and complete kits.

[0099] In certain embodiments, a collection of objects comprises elements and consumable for circuit board assembly and electronic device repair kits. In certain embodiments, a task comprises assembly and / or repair of electronic equipment (e.g., circuit boards), e.g., preparation of component trays for circuit board assembly or device repairs. The importance of the assembly process in this case is associated with reducing the risk of errors in highly sensitive operations by providing precisely ordered components and tools.

[0100] In certain embodiments, a collection of objects comprises instruments and consumables for beauty and skincare treatments and kits. In certain embodiments, a task comprises a beauty and / or skincare treatment, e.g., preparation of trays for beauty or skincare treatments. The importance of the assembly process in this case is associated with ensuring hygiene, proper sequencing of products, and a smooth workflow for practitioners.

[0101] In certain embodiments, a collection of objects comprises elements and consumable of emergency response kits and trauma trays (e.g., for ambulances). In certain Page 27 of 5713305099vlAttorney Docket No. : 2018997-0010embodiments, a task comprises a medical emergency-response procedure, e.g., assembly of emergency response kits or trauma trays for ambulances. The importance of the assembly process in this case is associated with providing quick access to necessary medical equipment in high-pressure situations, potentially saving lives.5. Illustrative Software, Computing Devices, and Environments

[0102] Certain embodiments described herein make use of computer algorithms in the form of software instructions executed by a computer processor. In certain embodiments, the software instructions include a machine learning (ML) module, also referred to herein as artificial intelligence (Al) software. As used herein, a machine learning module refers to a computer implemented process (e.g., a software function) that implements one or more specific machine learning techniques, e.g., artificial neural networks (ANNs), e.g., convolutional neural networks (CNNs), random forest, decision trees, support vector machines, and the like, in order to determine, for a given input, one or more output values. In certain embodiments, the input comprises image data and / or alphanumeric data which can include 2D and / or 3D datasets, numbers, words, phrases, or lengthier strings, for example. In certain embodiments, the one or more output values comprise image data (e.g., 2D and / or 3D datasets) and / or values representing numeric values, words, phrases, or other alphanumeric strings.

[0103] In certain embodiments, machine learning modules implementing machine learning techniques are trained, for example, using datasets that include categories of data described herein. Such training may be used to determine various parameters of machine learning algorithms implemented by a machine learning module, such as weights associated with layers in neural networks. In certain embodiments, once a machine learning module is trained, e.g., to accomplish a specific task such as identifying certain response strings, values of determined parameters are fixed and the (e.g., unchanging, static) machine learning module is used to process new data (e.g., different from the training data) and accomplish its trained task without further updates to its parameters (e.g., the machine learning module does not receive feedback and / or updates). In certain embodiments, available input data includes training data and validation data, e.g., where the validation data is separate and non-overlapping with the training data. For example, in certain embodiments, training data is used during the trainingPage 28 of 5713305099vlAttorney Docket No. : 2018997-0010process to optimize a model, whereas validation data is used to check the accuracy of the model while operating on previously unseen data. In certain embodiments, training data is divided into batches (e.g., portions) that is sequentially used (e.g., in random order) as sets of inputs to train a model. In certain embodiments, a model is trained multiple times (e.g., epochs) on the entire set of training data. In certain embodiments, machine learning modules may receive feedback, e.g., based on user review of accuracy, and such feedback may be used as additional training data, to dynamically update the machine learning module. In certain embodiments, two or more machine learning modules may be combined and implemented as a single module and / or a single software application. In certain embodiments, two or more machine learning modules may also be implemented separately, e.g., as separate software applications. A machine learning module may be software and / or hardware. For example, a machine learning module may be implemented entirely as software, or certain functions of a ANN module may be carried out via specialized hardware (e.g., via an application specific integrated circuit (ASIC) and / or field programmable gate arrays (FPGAs)).

[0104] In certain embodiments, machine learning modules implementing machine learning techniques may be composed of individual nodes (e.g., units, neurons). A node may receive a set of inputs that may include at least a portion of a given input data for the machine learning module and / or at least one output of another node. A node may have at least one parameter to apply and / or a set of instructions to perform (e.g., mathematical functions to execute) over the set of inputs. In certain embodiments, node instructions may include a step to provide various relative importance to the set of inputs using various parameters, such as weights. The weights may be applied by performing scalar multiplication (e.g., or other mathematical function) between a set of inputs values and the parameters, resulting in a set of weighted inputs. In certain embodiments, a node may have a transfer function to combine the set of weighted inputs into one output value. A transfer function may be implemented by a summation of all the weighted inputs and the addition of an offset (e.g., bias) value. In certain embodiments, a node may have an activation function to introduce non-linearity into the output value. Non-limiting examples of the activation function include Rectified Linear Activation (ReLu), logistic (e.g., sigmoid), hyperbolic tangent (tanh), and softmax. In certain embodiments, a node may have a capability of remembering previous states (e.g., recurrent nodes). Previous states may be applied to the input and output values using a set of learning parameters.Page 29 of 5713305099vlAttorney Docket No. : 2018997-0010

[0105] In certain embodiments, the machine learning module comprises a deep learning architecture composed of nodes organized into layers. For example, a layer is a set of nodes that receives data input (e.g., weighted or non- weighted input), transforms it (e.g., by carrying out instructions, e.g., applying a set of functions e.g., linear and / or non-linear functions), and passes transformed values as output (e.g., to the next layer). In certain embodiments, the set of nodes in a particular layer may share the same parameters and instructions without interacting with each other. A machine learning module may be composed of at least one layer (e.g., ordered).Examples of types of layers include convolutional layers (e.g., layers with a kernel, a matrix of parameters that is slid across an input to be multiplied with multiple input values to reduce them to a single output value); fully connected (FC) layers (e.g. all nodes are connected to all outputs of the previous layer); recurrent layers, long / short term memory (LSTM) layers, gated recurrent unit (GRU) layers (e.g., nodes with the various abilities to memorize and apply their previous inputs and / or outputs); batch normalization (BN) layers (e.g., layers that normalize a set of outputs from another layer, allowing for more independent learning of individual layers); activation layers (e.g., layers with nodes that only contain an activation function); and / or (un)pooling layers [e.g., layers that reduce (increase) dimensions of an input by summarizing (splitting) input values in defined patches).

[0106] In certain embodiments, the performance of a machine learning module may be characterized by its ability to produce an output data with specific accuracy. To achieve specific accuracy, a training process is performed to find optimal parameters, such as weights, for each node in each layer of the machine learning module. In certain embodiments, the training process of a machine learning module may involve using output data to calculate an objective function (e.g., cost function, loss function, error function) that needs to be optimized (e.g., minimized, maximized). For example, a machine learning objective function may be a combination of a loss function and regularization parameter. The loss function is related to how well the output is able to predict the input. The loss function may take various forms, like mean squared error, mean absolute error, binary cross-entropy, categorical cross-entropy, for example. The regularization term may be needed to prevent overfitting and improve generalization of the training process. Examples of regularization techniques include LI Regularization or Lasso Regression, L2 Regularization or Ridge Regression, and Dropout (e.g., dropping layer outputs at random during training process).Page 30 of 5713305099vlAttorney Docket No. : 2018997-0010

[0107] In certain embodiments, objective function optimization of a machine learning module may involve finding at least one (e.g., all) of the present global optima (e.g., as opposed to local optima). In certain embodiments, the algorithm for objective function optimization follows principles of mathematical optimization for a multi-variable function and relies on achieving specific accuracy of the process. Examples of objective function optimization algorithms include gradient descent, nonlinear conjugate gradient, random search, Levenberg-Marquardt algorithm, limited-memory Broyden-Fietcher-Goldfarb-Shanno algorithm, pattern search, basin hopping method, Krylov method, Adam method, genetic algorithm, particle swarm optimization, surrogate optimization, and simulated annealing.

[0108] In certain embodiments, the machine learning modules comprise one of more generative Al modules. Rather than depending on use of predetermined weights and rules, generative Al leverages complex neural networks and algorithms to understand patterns and produce output that mimic human creativity. Examples of generative Al modules include image synthesis models (e.g., DALL-E3, DALL-E2, Imagen 3 in Gemini, Craiyon, and the like) and text generation models (e.g., ChatGPT, GPT-4, and the like).

[0109] In certain embodiments, the machine learning modules comprises one or more image-based segmentation neural networks. Illustrative examples of segmentation neural networks include, for instance, Deep Image Matting (DIM), Semantic Segmentation methods (U-Net, DeepLab Series), Mask R-CNN, Chroma Keying CNNs, RefineNet, and MODNet.

[0110] In certain embodiments, Al used to generate alphanumeric text responsive to a user query and / or a set of input data may comprise (and / or utilize) one or more large language models (LLMs) [e.g., wherein the one or more LLMs comprise(s) one or more members selected from the group consisting of: BERT (Google) (or other transformer-based models), Falcon 40B, Galactica, GPT-3 (Generative Pre-trained Transformer, OpenAI), GPT-3.5 (OpenAI), GPT-4 (OpenAI), LaMDA (language model for dialogue applications, Google), Llama (large language model Meta Al) (Meta), Orca LLM (Microsoft), PaLM (Pathways Language Model), Phi-1 (Microsoft), StableLM (Stability Al), BLOOM (Hugging Face), RoBERTa (Meta), XLM-RoBERTa (Meta), NeMO LLM (Nvidia), XLNet (Google), Generate (Cohere), GLM-130B (Hugging Face), and Claude (Anthropic)] [e.g., wherein the one or more LLMs comprise(s) one or more members selected from the group consisting of an autoregressive LLM, autoencoding LLM, encoder-decoder LLM, bidirectional LLM, Fine-tuned LLMs, and multimodal LLMs].Page 31 of 5713305099vlAttorney Docket No. : 2018997-0010

[0111] In some embodiments, one or more non-transitory computer readable storage media is encoded with a computer program, wherein the program includes instructions that when executed by one or more processors cause the one or more processors to perform operations to perform a method of the present disclosure.

[0112] In some embodiments, a computer system includes a memory and one or more processors coupled to the memory, wherein the one or more processors are configured to perform a method of the present disclosure.

[0113] Illustrative embodiments of systems and methods disclosed herein are described with reference to determinations and / or models that may be performed or used by a computing device. That is, in some embodiments, methods disclosed herein are computer-implemented methods and, in some embodiments, a system as disclosed herein includes a processor and one or more non-transitory computer readable storage media (e.g., one or more memories) that have instructions stored thereon that, when executed by the processor, cause the processor to perform operations that include a method disclosed herein. For example, a system for supporting assembly of a collection of objects to be used for a task may be stored on a memory. Such a system may be used by a processor to perform a method. Methods of the present disclosure, or portions thereof, may be performed using a processor. The processor may be a part of a computing device and / or computing system.

[0114] Systems of the present disclosure may include a processor and / or a memory. The memory may store one or more programs that include instructions that when executed by a processor cause at least a portion of a method disclosed herein to be performed. The system may further include a machine-learned model. Additionally or alternatively, a remotely stored and / or operated machine-learned model may be accessed by a (e.g., the) processor. The processor and / or memory may be a part of a computing device and / or computing system.

[0115] One or more non-transitory computer readable media may store one or more programs that include instructions that when executed by a (e.g., the) processor cause at least a portion of a method disclosed herein to be performed.

[0116] As shown in FIG. 8, an implementation of a network environment 800 for use in providing systems, methods, and architectures as described herein is shown and described. In brief overview, referring now to FIG. 8, a block diagram of an exemplary cloud computing environment 800 is shown and described. The cloud computing environment 800 may include Page 32 of 5713305099vlAttorney Docket No. : 2018997-0010one or more resource providers 802a, 802b, 802c (collectively, 802). Each resource provider 802 may include computing resources. In some implementations, computing resources may include any hardware and / or software used to process data. For example, computing resources may include hardware and / or software capable of executing algorithms, computer programs, and / or computer applications. In some implementations, exemplary computing resources may include application servers and / or databases with storage and retrieval capabilities. Each resource provider 802 may be connected to any other resource provider 802 in the cloud computing environment 800. In some implementations, the resource providers 802 may be connected over a computer network 808. Each resource provider 802 may be connected to one or more computing device 804a, 804b, 804c (collectively, 804), over the computer network 808.

[0117] The cloud computing environment 800 may include a resource manager 806. The resource manager 806 may be connected to the resource providers 802 and the computing devices 804 over the computer network 808. In some implementations, the resource manager 806 may facilitate the provision of computing resources by one or more resource providers 802 to one or more computing devices 804. The resource manager 806 may receive a request for a computing resource from a particular computing device 804. The resource manager 806 may identify one or more resource providers 802 capable of providing the computing resource requested by the computing device 804. The resource manager 806 may select a resource provider 802 to provide the computing resource. The resource manager 806 may facilitate a connection between the resource provider 802 and a particular computing device 804. In some implementations, the resource manager 806 may establish a connection between a particular resource provider 802 and a particular computing device 804. In some implementations, the resource manager 806 may redirect a particular computing device 804 to a particular resource provider 802 with the requested computing resource.

[0118] FIG. 9 shows an example of a computing device 900 and a mobile computing device 950 that can be used to implement the techniques described in this disclosure. The computing device 900 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The mobile computing device 950 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, and other similar computing devices. The components shown here, their connections Page 33 of 5713305099vlAttorney Docket No. : 2018997-0010and relationships, and their functions, are meant to be examples only, and are not meant to be limiting.

[0119] The computing device 900 includes a processor 902, a memory 904, a storage device 906, a high-speed interface 908 connecting to the memory 904 and multiple high-speed expansion ports 910, and a low-speed interface 912 connecting to a low-speed expansion port 914 and the storage device 906. Each of the processor 902, the memory 904, the storage device 906, the high-speed interface 908, the high-speed expansion ports 910, and the low-speed interface 912, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor 902 can process instructions for execution within the computing device 900, including instructions stored in the memory 904 or on the storage device 906 to display graphical information for a GUI on an external input / output device, such as a display 916 coupled to the high-speed interface 908. In other implementations, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system). Thus, as the term is used herein, where a plurality of functions are described as being performed by “a processor”, this encompasses embodiments wherein the plurality of functions are performed by any number of processors (one or more) of any number of computing devices (one or more). Furthermore, where a function is described as being performed by “a processor”, this encompasses embodiments wherein the function is performed by any number of processors (one or more) of any number of computing devices (one or more) (e.g., in a distributed computing system).

[0120] The memory 904 stores information within the computing device 900. In some implementations, the memory 904 is a volatile memory unit or units. In some implementations, the memory 904 is a non-volatile memory unit or units. The memory 904 may also be another form of computer-readable medium, such as a magnetic or optical disk.

[0121] The storage device 906 is capable of providing mass storage for the computing device 900. In some implementations, the storage device 906 may be or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. Instructions can be stored in Page 34 of 5713305099vlAttorney Docket No. : 2018997-0010an information carrier. The instructions, when executed by one or more processing devices (for example, processor 902), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices such as computer- or machine-readable mediums (for example, the memory 904, the storage device 906, or memory on the processor 902).

[0122] The high-speed interface 908 manages bandwidth-intensive operations for the computing device 900, while the low-speed interface 912 manages lower bandwidth-intensive operations. Such allocation of functions is an example only. In some implementations, the highspeed interface 908 is coupled to the memory 904, the display 916 (e.g., through a graphics processor or accelerator), and to the high-speed expansion ports 910, which may accept various expansion cards (not shown). In the implementation, the low-speed interface 912 is coupled to the storage device 906 and the low-speed expansion port 914. The low-speed expansion port 914, which may include various communication ports (e.g., USB, Bluetooth®, Ethernet, wireless Ethernet) may be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

[0123] The computing device 900 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server 920, or multiple times in a group of such servers. In addition, it may be implemented in a personal computer such as a laptop computer 922. It may also be implemented as part of a rack server system 924. Alternatively, components from the computing device 900 may be combined with other components in a mobile device (not shown), such as a mobile computing device 950. Each of such devices may contain one or more of the computing device 900 and the mobile computing device 950, and an entire system may be made up of multiple computing devices communicating with each other.

[0124] The mobile computing device 950 includes a processor 952, a memory 964, an input / output device such as a display 954, a communication interface 966, and a transceiver 968, among other components. The mobile computing device 950 may also be provided with a storage device, such as a micro-drive or other device, to provide additional storage. Each of the processor 952, the memory 964, the display 954, the communication interface 966, and thePage 35 of 5713305099vlAttorney Docket No. : 2018997-0010transceiver 968, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

[0125] The processor 952 can execute instructions within the mobile computing device 950, including instructions stored in the memory 964. The processor 952 may be implemented as a chipset of chips that include separate and multiple analog and digital processors. The processor 952 may provide, for example, for coordination of the other components of the mobile computing device 950, such as control of user interfaces, applications run by the mobile computing device 950, and wireless communication by the mobile computing device 950.

[0126] The processor 952 may communicate with a user through a control interface 958 and a display interface 956 coupled to the display 954. The display 954 may be, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display) display or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface 956 may comprise appropriate circuitry for driving the display 954 to present graphical and other information to a user. The control interface 958 may receive commands from a user and convert them for submission to the processor 952. In addition, an external interface 962 may provide communication with the processor 952, so as to enable near area communication of the mobile computing device 950 with other devices. The external interface 962 may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.

[0127] The memory 964 stores information within the mobile computing device 950. The memory 964 can be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. An expansion memory 974 may also be provided and connected to the mobile computing device 950 through an expansion interface 972, which may include, for example, a SIMM (Single In Line Memory Module) card interface. The expansion memory 974 may provide extra storage space for the mobile computing device 950, or may also store applications or other information for the mobile computing device 950. Specifically, the expansion memory 974 may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, the expansion memory 974 may be provide as a security module for the mobile computing device 950, and may be programmed with instructions that permit secure use of the mobile computing device 950. In addition, secure applications may be provided via the Page 36 of 5713305099vlAttorney Docket No. : 2018997-0010SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.

[0128] The memory may include, for example, flash memory and / or NVRAM memory (non-volatile random access memory), as discussed below. In some implementations, instructions are stored in an information carrier. The instructions, when executed by one or more processing devices (for example, processor 952), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices, such as one or more computer- or machine-readable mediums (for example, the memory 964, the expansion memory 974, or memory on the processor 952). In some implementations, the instructions can be received in a propagated signal, for example, over the transceiver 968 or the external interface 962.

[0129] The mobile computing device 950 may communicate wirelessly through the communication interface 966, which may include digital signal processing circuitry where necessary. The communication interface 966 may provide for communications under various modes or protocols, such as GSM voice calls (Global System for Mobile communications), SMS (Short Message Service), EMS (Enhanced Messaging Service), or MMS messaging (Multimedia Messaging Service), CDMA (code division multiple access), TDMA (time division multiple access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), among others. Such communication may occur, for example, through the transceiver 968 using a radio-frequency. In addition, short-range communication may occur, such as using a Bluetooth®, Wi-Fi™, or other such transceiver (not shown). In addition, a GPS (Global Positioning System) receiver module 970 may provide additional navigation- and location-related wireless data to the mobile computing device 950, which may be used as appropriate by applications running on the mobile computing device 950.

[0130] The mobile computing device 950 may also communicate audibly using an audio codec 960, which may receive spoken information from a user and convert it to usable digital information. The audio codec 960 may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of the mobile computing device 950. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, etc.) and may also include sound generated by applications operating on the mobile computing device 950.Page 37 of 5713305099vlAttorney Docket No. : 2018997-0010

[0131] The mobile computing device 950 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone 980. It may also be implemented as part of a smart-phone 982, personal digital assistant, or other similar mobile device.

[0132] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0133] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms machine-readable medium and computer-readable medium refer to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term machine-readable signal refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0134] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.Page 38 of 5713305099vlAttorney Docket No. : 2018997-0010

[0135] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0136] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0137] It is contemplated that systems, devices, methods, and processes of the disclosure encompass variations and adaptations developed using information from the embodiments described herein. Adaptation and / or modification of the systems, devices, methods, and processes described herein may be performed by those of ordinary skill in the relevant art.

[0138] Throughout the description, where articles, devices, and systems are described as having, including, or comprising specific components, or where processes and methods are described as having, including, or comprising specific steps, it is contemplated that, additionally, there are articles, devices, and systems according to certain embodiments of the present disclosure that consist essentially of, or consist of, the recited components, and that there are processes and methods according to certain embodiments of the present disclosure that consist essentially of, or consist of, the recited processing steps.

[0139] It should be understood that the order of steps or order for performing certain action is immaterial so long as operability is not lost. Moreover, two or more steps or actions may be conducted simultaneously. As is understood by those skilled in the art, the terms “over”, “under”, “above”, “below”, “beneath”, and “on” are relative terms and can be interchanged in reference to different orientations of the layers, elements, and substrates included in the presentPage 39 of 5713305099vlAttorney Docket No. : 2018997-0010disclosure. For example, a first layer on a second layer, in some embodiments means a first layer directly on and in contact with a second layer. In other embodiments, a first layer on a second layer can include another layer there between.

[0140] Headers are provided for the convenience of the reader and are not intended to be limiting with respect to the claimed subject matter.

[0141] Certain embodiments of the present disclosure were described above. It is, however, expressly noted that the present disclosure is not limited to those embodiments, but rather the intention is that additions and modifications to what was expressly described in the present disclosure are also included within the scope of the disclosure. Moreover, it is to be understood that the features of the various embodiments described in the present disclosure were not mutually exclusive and can exist in various combinations and permutations, even if such combinations or permutations were not made express, without departing from the spirit and scope of the disclosure. The disclosure has been described in detail with particular reference to certain embodiments thereof, but it will be understood that variations and modifications can be effected within the spirit and scope of the claimed invention.Page 40 of 5713305099vl

Claims

Attorney Docket No. : 2018997-0010What is claimed is:

1. A method for supporting assembly of a collection of objects (e.g., surgical instruments) to be used for a task (e.g., a surgical procedure), the method comprising:receiving, by a processor of a computing device, images [e.g., as a video stream (e.g., a real time video stream)] from (e.g., directly from) a camera;automatically identifying and / or tracking, by the processor, objects discernable (e.g., segmentable and, optionally, classifiable) in the images;automatically determining, by the processor, that the objects have been placed in a container (e.g., a tray); andupdating, by the processor, a container inventory for the container corresponding to a task to indicate that the objects have been placed in the container (e.g., are in the container).

2. The method of claim 1, wherein the identifying and / or tracking objects and the determining that the objects have been placed in the container are each performed, by the processor, using the images.

3. The method of claims 1 or 2, wherein the container is discernable in the images.

4. The method of claims 1 or 2, wherein the determining that the objects have been placed in the container comprises determining that the objects have moved to a region.

5. The method of claim 4, wherein the region is discernible in the images.

6. The method of claim 4, wherein the region is outside of a field of view of the images.

7. The method of any one of claims 1-6, wherein the determining that the objects have been placed in the container comprises determining, by the processor, that the objects are present in a region (e.g., area or volume) (e.g., with a computer vision module).Page 41 of 5713305099vlAttorney Docket No. : 2018997-00108. The method of claim 7, wherein the objects are individually determined to be present in the region.

9. The method of claims 7 or 8, comprising:automatically determining, by the processor, the region based on a position of the container in the images.

10. The method of claim 9, wherein the determining the region comprises segmenting and classifying, by the processor, the container (e.g., with a computer vision module).

11. The method of any one of claims 7-10, wherein the region is a first region and the method comprises, for each of the objects, automatically identifying, by the processor, that the object is present in a second region outside of the first region.

12. The method of any one of claims 7-11, wherein the region is a first region and the method comprises, for each of the objects, automatically identifying, by the processor, the object when the object is present in a second region outside of the first region.

13. The method of any one of claims 1-12, wherein the determining that the objects have been placed in the container comprises tracking, by the processor, (e.g., with a machine vision module) the objects to determine that the objects have moved from the second region to the first region.

14. The method of any one of claims 1-13, wherein the determining that the objects have been placed in the container comprises individually determining, by the processor, that each of the objects has been placed in the container.

15. The method of any one of claims 1-14, wherein the updating the container inventory comprises updating, by the processor, the container inventory as each of the objects has been placed in the container.Page 42 of 5713305099vlAttorney Docket No. : 2018997-001016. The method of any one of claims 1-15, comprising:determining, by the processor, from the images that one or more of the objects have been removed from the container; andupdating, by the processor, the container inventory to indicate that the one or more of the objects is not in the container.

17. The method of any one of claims 1-16, comprising:automatically determining, by the processor, that the container inventory is complete based on a determination that all of the objects are being disposed in the container at a same time.

18. The method of claim 17, comprising:automatically providing, by the processor, a notification (e.g., rendering and displaying a graphic or providing a sound) to a user in response to determining that the container inventory is complete.

19. The method of any one of claims 1-18, comprising:providing, by the processor, a notification (e.g., rendering and displaying a graphic or providing a sound) to a user as each of the objects is determined to be placed in the container.

20. The method of any one of claims 1-19, wherein the objects are individually determined to be placed in the container.

21. The method of any one of claims 1-20, wherein the container is discernible in the images.

22. The method of any one of claims 1-21, wherein the camera comprises one or more cameras.

23. The method of any one of claims 1-22, wherein the camera comprises sensitivity to at least a part of a spectrum of visible light (e.g., infrared, near-infrared, ultra-violet).Page 43 of 5713305099vlAttorney Docket No. : 2018997-001024. The method of any one of claims 1-23, wherein the camera comprises an exposure time such that the objects are discernible in the images (e.g., not blurred).

25. The method of any one of claims 1-24, wherein the camera comprises a focal length and / or a focal depth such that the objects remain in focus.

26. The method of any one of claims 1-25, comprising obtaining the images with the camera.

27. The method of any one of claims 1-25, wherein the images have been obtained as a video stream (e.g., a real time video stream).

28. The method of any one of claims 1-25, comprising:obtaining the images using the camera that is disposed to capture the first region and / or the second region.

29. The method of any one of claim 1-28, comprising showing (e.g., displaying, e.g., by a user, e.g., in the first and / or second region) (e.g., individually) each of the objects to the camera (e.g., individually) [e.g., in a region outside of (e.g., adjacent to) a container] (e.g., a second region).

30. The method of claim 29, wherein the showing comprises presenting each of the objects (e.g., individually) to the camera in a field of view of the camera.

31. The method of claim 30, wherein each of the objects is at least 80% (e.g., 50%, 60%, 70%, 90%, 95%) unobstructed during the presenting.

32. The method of any one of claims 1-31, wherein the identifying and / or tracking objects comprises segmenting and / or classifying, by the processor, (e.g., with a machine vision module) the objects in the images.Page 44 of 5713305099vlAttorney Docket No. : 2018997-001033. The method of claim 32, wherein the classifying the objects in the images comprises determining, for each of the objects, by the processor, whether the object is to be placed into the container according to the container inventory (e.g., whether the object identifies as an entry in the container inventory or not, whether a number of objects of a same kind as the object have been already placed in the container).

34. The method of any one of claims 1-33, wherein the container inventory comprises entries that specify, for each of the objects, the object and a number of the object to be placed in the container, wherein each entry in the list comprises a type of object (e.g., a name, an identification) and a quantity (e.g., a remaining quantity of objects of a same type to be placed in the container),wherein the identifying and / or tracking objects discernable in the images comprises, for each of the objects, determining an entry in the list associated with the object (e.g., by classifying the object using the container inventory), andwherein the updating the container inventory comprises, for each of the objects, updating the quantity of the entry of the object.

35. The method of any one of claims 1-34, wherein the identifying and / or tracking comprises identifying the objects.

36. The method of claim 35, wherein the identifying is performed with a machine vision module.

37. The method of any one of claims 1-34, wherein the identifying and / or tracking comprises tracking the objects.

38. The method of claim 37, wherein the tracking is performed with a machine vision module.

39. The method of any one of claims 1-38, wherein the determining that the objects have been placed in the container is performed with a machine vision module.Page 45 of 5713305099vlAttorney Docket No. : 2018997-001040. The method of any one of claims 36-39, wherein the machine vision module comprises an image classification module.

41. The method of any one of claims 36-39, wherein the machine vision module comprises a semantic segmentation module.

42. The method of any one of claims 36-39, wherein the machine vision module comprises an instance segmentation module.

43. The method of any one of claims 36-39, wherein the machine vision module comprises an image classification with localization module.

44. The method of any one of claims 36-39, wherein the machine vision module comprises an object recognition module.

45. The method of any one of claims 36-39, wherein the machine vision module comprises an object detection module.

46. The method of any one of claims 36-39, wherein the machine vision module comprises a pattern recognition module.

47. The method of any one of claims 36-39, wherein the machine vision module comprises an edge detection module.

48. The method of any one of claims 36-39, wherein the machine vision module comprises a feature matching module.

49. The method of any one of claims 36-39, wherein the machine vision module comprises a deep learning module.Page 46 of 5713305099vlAttorney Docket No. : 2018997-001050. The method of any one of claims 1-49, wherein the determining that the objects have been placed in the container comprises:receiving, by the processor, a weight for the container; andverifying, by the processor, contents of the container are correct based on the weight.

51. The method of any one of claims 1-50, comprising receiving, by the processor, a weight for the container, wherein the determining that the objects have been placed in the container is based on the weight.

52. The method of any one of claims 1-51, comprisingreceiving, by the processor, a weight for the container; anddetermining, by the processor, that one or more of the objects have been placed in the container based on the weight.

53. The method of any one of claims 1-52, comprising:receiving, by the processor, a weight for the container; andupdating, by the processor, the container inventory based on the weight.

54. The method of any one of claims 1-53, comprising:receiving, by the processor, a weight for the container; anddetermining, by the processor, that the weight is incorrect for one or more of the objects that have been placed in the container.

55. The method of claim 54, comprising updating, by the processor, the container inventory (e.g., to remove an object) based on the weight.

56. The method of claim 54, comprising preventing (e.g., stopping), by the processor, an update of the container inventory based on the weight [e.g., that would have otherwise been updated based on the images (e.g., based on identifying and / or tracking of objects discernible therein)].Page 47 of 5713305099vlAttorney Docket No. : 2018997-001057. The method of any one of claims 50-53, comprising weighing the container with a balance to determine the weight.

58. The method of claim 57, wherein the container is disposed on the balance throughout a period of time during which the images were (e.g., or are) acquired.

59. The method of any one of claims 1-58, wherein the container comprises a tag (e.g., a QR code, barcode, RFID tag) (e.g., a printed tag) (e.g., a marking) (e.g., disposed on an upward facing surface of the container) and the method comprises selecting, by the processor, the container inventory based on the tag.

60. The method of claim 59, wherein selecting the container inventory comprises reading, by the processor, the tag using the images (e.g., using a machine vision module).

61. The method of claim 59, comprising providing, by the processor, information corresponding to the tag, wherein the container inventory is selected based on the information.

62. The method of claim 61, comprising obtaining the information.

63. The method of claim 62, wherein obtaining the information comprises scanning the tag.

64. The method of any one of claims 1-63, comprising providing, by the processor, a graphical user interface (GUI) (e.g., rendering and / or displaying the GUI on a screen) for user assistance (e.g., display the notification, receive user input, display the images).

65. The method of claim 64, wherein the GUI provides an automated, real-time, interactive assistance to the user in the assembling the collection of objects used for the task.

66. The method of claims 64 or 65, wherein the GUI comprises an inventory widget for displaying the container inventory and wherein the updating the container inventory comprisesPage 48 of 5713305099vlAttorney Docket No. : 2018997-0010updating the inventory widget and / or displaying one or more associated notifications in the inventory widget.

67. The method of any one of claims 64-66, wherein the GUI comprises an image widget for displaying the images and wherein the identifying and / or tracking objects comprises, for each of the objects, displaying a notification (e.g., as an overlay) in the image widget.

68. The method of any one of claims 64-67, wherein the GUI comprises an indication widget for displaying indication messages to the user and wherein the identifying and / or tracking objects comprises, for each of the objects, displaying the indication messages (e.g., a thumbnail, ID, and / or name associated with an object) in the indication widget.

69. The method of any one of claims 64-68, wherein the GUI comprises a tracking status widget for displaying status messages to the user and wherein the identifying and / or tracking objects and the determining that the objects have been placed in the container comprise, for each of the objects, displaying a status messages (e.g., “Tracking”, “Placed”, “Wrong Instrument”) in the tracking status widget.

70. The method of any one of claims 1-69, wherein the method is performed in real time.

71. The method of any one of claims 1-70, wherein the method is performed in real time as the images are acquired.

72. The method of any one of claims 1-71, wherein the method is performed in real time as a user assembles the objects in the container.

73. The method of any one of claims 1-72, wherein the method is performed as the objects are assembled into the container (e.g., in real time).

74. The method of any one of claims 1-73, wherein the objects are assembled in a sterile environment.Page 49 of 5713305099vlAttorney Docket No. : 2018997-001075. The method of any one of claims 1-74, wherein, upon determination that all of the objects are in the container, the objects are sterilized (e.g., in the container) (e.g., and packaged for sterile storage and / or use).

76. The method of any one of claims 1-75, wherein the collection of objects comprises surgical instruments.

77. The method of claim 76, comprising sterilizing the objects while in the container.

78. The method of any one of claims 1-75, wherein the collection of objects comprises meal components (e.g., foods, beverages).

79. The method of any one of claims 1-75, wherein the collection of objects comprises laboratory and research instruments and consumables.

80. The method of any one of claims 1-75, wherein the collection of objects comprises components of medication packaging.

81. The method of any one of claims 1-75, wherein the collection of objects comprises dentistry instruments and consumables.

82. The method of any one of claims 1-75, wherein the collection of objects comprises tools and kits for vehicle manufacturing and repair workstations.

83. The method of any one of claims 1-75, wherein the collection of objects comprises elements of medical kits and repair kits.

84. The method of any one of claims 1-75, wherein the collection of objects comprises elements and consumable for circuit board assembly and electronic device repair kits.Page 50 of 5713305099vlAttorney Docket No. : 2018997-001085. The method of any one of claims 1-75, wherein the collection of objects comprises instruments and consumables for beauty and skincare treatments and kits.

86. The method of any one of claims 1-75, wherein the collection of objects comprises elements and consumable of emergency response kits and trauma trays (e.g., for ambulances).

87. The method of any one of claims 1-75, wherein the task comprises a surgical procedure.

88. The method of any one of claims 1-75, wherein the task comprises food consumption.

89. The method of any one of claims 1-75, wherein the task comprises an experimental (e.g., laboratory, research) procedure.

90. The method of any one of claims 1-75, wherein the task comprises pharmaceutical manufacturing (e.g., packaging).

91. The method of any one of claims 1-75, wherein the task comprises a dental procedure.

92. The method of any one of claims 1-75, wherein the task comprises vehicle manufacturing and / or vehicle repair.

93. The method of any one of claims 1-75, wherein the task comprises assembly and / or repair of electronic equipment (e.g., circuit boards).

94. The method of any one of claims 1-75, wherein the task comprises a beauty and / or skincare treatment.

95. The method of any one of claims 1-75, wherein the task comprises a medical emergencyresponse procedure.Page 51 of 5713305099vlAttorney Docket No. : 2018997-001096. A method for supporting assembly of a collection of objects (e.g., surgical instruments) to be used for a task (e.g., a surgical procedure), the method comprising:receiving, by a processor of a computing device, a weight of a container and images from a camera in which one or more objects are discernable; anddetermining, by the processor, that the one or more objects have been placed in the container based on at least the images and the weight.

97. The method of claim 96, wherein determining that the one or more objects have been placed in the container based on the images comprises identifying and / or tracking the one or more objects using the images.

98. The method of claim 97, wherein the identifying and / or tracking comprises segmenting the one or more objects in the images.

99. The method of claim 98, wherein the identifying and / or tracking comprises classifying the one or more objects in the images.

100. The method of any one of claims 96-99, comprising updating, by the processor, a container inventory for the container corresponding to a task to indicate that the one or more objects have been placed in the container (e.g., are in the container).

101. A computer-implemented method for supporting assembly of a collection of objects to be used for a task, the method comprising, via one or more graphical user interfaces (GUIs):rendering and / or displaying on a screen, by the processor, one or more graphical user interface (GUIs), said one or more GUIs comprising an image widget for displaying a real time (e.g., live) stream of images (e.g., video stream);receiving, by the processor, a determination that an object discernible in the images has been identified and / or tracked; andin response to receiving the determination, providing (e.g., rendering and / or displaying) an indication, by the processor, in the one or more GUIs that the object belongs in a collection of objects used for a task.Page 52 of 5713305099vlAttorney Docket No. : 2018997-0010102. The method of claim 101, comprising, for each of the objects, rendering and / or displaying a notification that indicates when the object has been identified and / or tracked.

103. The method of claim 102, wherein the notification is rendered and / or displayed in the image widget.

104. The method of claim 103, wherein the notification is rendered and / or displayed as an overlay over the object in the stream of images.

105. The method of any one of claims 101-104, comprising providing (e.g., rendering and / or displaying), by the processor, an indication in the one or more GUIs that the objects have been placed in the container (e.g., are in the container) [e.g., as the objects are placed in the container (e.g., individually)] (e.g., an individual respective indication).

106. The method of claim 105, wherein the one or more GUIs comprise an inventory widget for displaying the container inventory and wherein the providing an indication comprises, for each of the objects, rendering and / or displaying a notification to the user in the inventory widget.

107. The method of any one of claims 101-106, wherein the one or more GUIs comprise a message widget for displaying a feedback message to the user and wherein the method comprises displaying the feedback message (e.g., about system performance, frame rate, connectivity to the camera, guidance instructions) in the message widget.

108. The method of any one of claims 101-107, wherein the one or more GUIs comprise an indication widget for displaying indication messages to the user and the method comprises, for each of the objects, displaying the indication messages (e.g., a thumbnail, ID, and / or name associated with an object) in the indication widget based on the determination received.

109. The method of any one of claims 101-108, wherein the one or more GUIs comprise a tracking status widget for displaying status messages to the user and wherein the method Page 53 of 5713305099vlAttorney Docket No. : 2018997-0010comprises, for each of the objects, displaying a status messages (e.g., “Tracking”, “Placed”, “Wrong Instrument”) in the tracking status widget.

110. The method of any one of claims 101-109, comprising:rendering and / or displaying on a screen, by the processor, a controls widget (e.g., a dialog box, drop down menu, radio button list) for entry of user display preferences (e.g., images, indications, notification preferences, organization of widgets in the one or more GUIs);receiving, by the processor, a selection from the user display preferences; and rendering and / or display on a screen, by the processor, the one or more GUIs according to the selection from the user display preference.

111. A computer-implemented method for supporting assembly of a collection of objects to be used for a task, the method comprising, via one or more graphical user interfaces (GUIs):rendering and / or displaying on a screen, by the processor, one or more graphical user interface (GUIs), said one or more GUIs comprising an inventory widget for displaying the container inventory;receiving, by the processor, a determination that an object discernible in the images has been identified and / or tracked; andin response to receiving the determination, providing (e.g., rendering and / or displaying) an indication, by the processor, in the GUI that the object belongs in a collection of objects used for a task.

112. The method of claims 111, wherein the providing an indication comprises, for each of the objects, displaying a notification to the user in the inventory widget.

113. The method of claims 111 or 112, wherein the one or more GUIs comprise an image widget for displaying the images and wherein the method comprises, for each object, rendering and / or displaying a notification that indicates when the object has been identified and / or tracked in the image widget.Page 54 of 5713305099vlAttorney Docket No. : 2018997-0010114. A system for supporting assembly of a collection of objects to be used for a task, the system comprising:a computing device comprising a processor and a memory having instructions stored thereon that, when executed by the processor, cause the processor to perform operations comprising the method according to any one of claims 1-113.

115. One or more non-transitory computer readable media having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform operations comprising the method of any one of claims 1-113.

116. The system of claim 114, comprising a camera.

117. A system for supporting assembly of a collection of objects to be used for a task, the system comprising:a camera for collecting images;a computing device for automatically identifying and / or tracking objects discernable in the images to determine when the objects are placed in a container.

118. The system of claim 117, wherein the computing device comprises a computer vision module that performs the identifying and / or tracking.

119. The system of claims 117 or 118, wherein the camera comprises one or more cameras.

120. The system of any one of claims 117-119, wherein the camera is an overhead camera.

121. The system of any one of claims 117-120, wherein the camera is disposed such that a field of view of the camera comprises a workspace for assembling the collection of objects.

122. The system of any one of claims 117-121, wherein the camera comprises sensitivity to at least a part of a spectrum of visible light (e.g., infrared, near-infrared, ultra-violet).Page 55 of 5713305099vlAttorney Docket No. : 2018997-0010123. The system of any one of claims 117-122, wherein the camera comprises an exposure time such that the objects are discernible in the images (e.g., not blurred).

124. The system of any one of claims 117-123, wherein the camera comprises a focal length and / or a focal depth such that the objects remain in focus.

125. The system of any one of claims 117-124, wherein the camera is disposed such that a field of view of the camera comprises a region (e.g., volume, area) associated with the container.

126. The system of any one of claims 117-125, wherein the camera is disposed such that, for each of the objects, a field of view of the camera comprises at least 80% (e.g., 50%, 60%, 70%, 90%, 95%) of unobstructed view of the object (e.g., as measured by a projection area of the object in a particular position).

127. The system of any one of claims 117-126, the system comprising a balance for weighing the container and the objects.

128. The system of claim 127, wherein the container is disposed on the balance throughout a period of time during which the images were (e.g., or are) acquired.

129. The system of claims 127 or 128, wherein, for each of the objects, the identifying and / or tracking is based at least partly on a weight of the object determined by the balance.

130. The system of any one of claims 127-129, wherein, for each of the objects, the determination that the object is placed in the container is based at least partly on a weight of the object determined by the balance.

131. The system of any one of claims 127-130, the system comprising a container inventory.

132. The system of claim 131, wherein, for each of the objects, the container inventory is updated based on a weight of the object determined by the balance.Page 56 of 5713305099vl

Images