Systems and methods for context-aware guidance and assistance using augmented reality

Abstract

Systems and methods provide real-time guidance for a user performing a task by integrating sensor measurements with augmented-reality (AR) presentation. One or more sensors, including external sensors integrated with tools, devices, or packaging, generate measurement data associated with physical quantities. A processor receives the measurement data and, using spatial parameters corresponding to locations associated with the measurements, generates a virtual representation of at least a portion of the user's environment and one or more overlays indicating a status or guidance for one or more areas of the environment. An AR device displays the overlays in spatial registration with the environment to guide the user during task execution. In some embodiments, computer vision and machine-learning inference recognize objects, orientation, or task context to adapt the overlays and guidance.

Description

SYSTEMS AND METHODS FOR CONTEXT-AWARE GUIDANCE AND ASSISTANCEUSING AUGMENTED REALITYRELATED APPLICATION DATA

[0001] This application claims the benefit of priority of U.S. Provisional Patent Application Serial No. 63 / 734,871, filed December 17, 2024, and titled “Augmented Reality- Assisted Oral Hygiene System with Integrated Live Camera and AI-Driven Feedback”, U.S. Provisional Patent Application Serial No. 63 / 735,039, filed December 17, 2024, and titled “System and Method for Scent Detection, Visualization, and Analysis Using External Aroma Analyzers and Augmented Reality Glasses”, U.S. Provisional Patent Application Serial No. 63 / 735,431, filed December 18, 2024, and titled “AR Glasses with Al and Camera for Hair Color Visualization, Precision Application, and Guidance”, U.S. Provisional Patent Application Serial No. 63 / 736,130, filed December 19, 2024, and titled “Augmented Reality Glasses for Cleaning Guidance”, and U.S. Provisional Patent Application Serial No. 63 / 737,256, filed December 20, 2024, and titled “Augmented Reality Glasses and Systems for Visualizing Tool, Product, and Application Interactions”, and U.S. Provisional Patent Application Serial No. 63 / 738,544, filed on December 24, 2024, and titled “System and Method for Wireless Integration of External Sensors with Augmented Reality Glasses for Application-Specific Guidance” each of which is incorporated by reference herein in its entirety.FIELD OF THE DISCLOSURE

[0002] The present disclosure generally relates to augmented reality devices with integrated sensors and in particular to systems and methods for providing application-specific guidance to augmented reality devices based on external sensors.BACKGROUND

[0003] Users across a wide range of applications face challenges that affect the optimal use of products and tools. For example, in applications such as personal care, users often struggle with ensuring even application of products such as hair dye, lotions, or creams, which may lead to uneven results or unnecessary product waste. Similarly, in cleaning applications, missed areas or overuse of cleaning solutions are common due to limited visibility, inconsistent technique, or the absence of1Attorney Docket No. 19842-006WOU1real-time guidance during task execution. In cooking applications, inconsistent proportions or uneven application of ingredients, such as oils or spices, may likewise result in variable or undesired outcomes when precise, context-aware feedback is unavailable at the time of use.

[0004] More generally, many existing approaches to addressing such challenges rely on instructional or visual guidance that is separate from the physical execution of the task. These approaches typically do not interact directly with the mechanisms that control material application, tool positioning, or force delivery, and therefore lack the ability to adapt dynamically to deviations in user motion or task conditions. As a result, even when guidance is provided, users may continue to experience inconsistent outcomes due to the absence of coordinated, real-time feedback during physical task execution.SUMMARY OF THE DISCLOSURE

[0005] The present disclosure relates to systems and methods for assisting users during execution of physical tasks by combining augmented reality (AR) guidance with sensing, inference, and, in certain embodiments, physical actuation. The disclosed systems provide spatially aligned virtual guidance that is registered to the user’s real -world environment and dynamically updated based on sensed physical conditions, thereby improving accuracy, consistency, safety, and task outcomes without requiring full automation.

[0006] In some embodiments, the system includes one or more sensing subsystems configured to detect physical features, surfaces, material states, environmental conditions, or tool behaviors associated with a task. Such sensing subsystems may include visual, depth, inertial, thermal, ultraviolet, magnetic, chemical, flow, force, or other sensors, including sensors integrated into tools, devices, packaging, or the surrounding environment. A processing subsystem analyzes sensor data, alone or in combination over time, to determine spatial and temporal relationships between detected physical states and one or more virtual guidance elements. In some embodiments, the processing subsystem reconstructs spatially and temporally varying distributions of physical quantities, including scalar or vector fields, from discrete sensor measurements, enabling visualization, inference, and guidance based on the evolution of such fields over time.

[0007] The system further includes an augmented reality display device configured to present virtual guidance overlays that are spatially registered to the physical environment as perceived by the2Attorney Docket No. 19842-006WOU1user. The virtual guidance overlays may indicate desired locations, regions, trajectories, orientations, coverage patterns, timing information, or predicted outcomes associated with the task, and may adapt dynamically based on sensor confidence, user behavior, environmental context, or inferred task state. In some embodiments, the system employs predictive modeling to anticipate user intent, material behavior, or task progression and to present proactive guidance prior to completion of a physical action.

[0008] In certain embodiments, the system includes one or more actuators operatively coupled to a tool, device, or instrument used by the user. The actuators may receive control signals derived from sensor data, inferred task parameters, or deviations between actual and desired physical states, and may influence physical characteristics such as position, orientation, force, speed, vibration, stiffness, illumination, or material delivery. Such actuation may be applied continuously, intermittently, or conditionally to assist task execution.

[0009] In some embodiments, the system operates in a shared-autonomy configuration in which the user retains primary control over gross motion, intent, or decision-making, while the system applies fine-scale corrective or assistive actions and corresponding AR feedback. The degree and form of assistance may vary dynamically based on task context, detected user skill level, environmental conditions, or confidence metrics associated with sensor data, allowing guidance to remain intuitive and non-intrusive.

[0010] The disclosed systems and methods may be applied across a wide range of applications, including but not limited to personal care, cleaning, material application, inspection, maintenance, manufacturing, medical procedures, diagnostics, and other tasks involving manual interaction with tools, materials, or environments. By integrating spatially registered AR guidance with multi-modal sensing, temporal analysis, and optional physical actuation in a coordinated framework, the disclosed embodiments enable users to perceive and interact with physical phenomena, including those that are invisible, transient, or difficult to sense, while preserving natural human control.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] For the purpose of illustrating the disclosed embodiments, the drawings show aspects thereof. It is to be understood, however, that the teachings of the present disclosure are not limited to the precise arrangements and instrumentalities shown in the drawings, wherein:3Attorney Docket No. 19842-006WOU1FIG. 1 depicts a system for detecting aromas in a region with a sensor and transmitting related data to AR glasses in accordance with an embodiment of the present disclosure;FIG. 2 illustrates an overlay shown in the field of view of a user’s AR glasses for the system shown in FIG. 1;FIG. 3 illustrates an overlay shown on a display for AR glasses indicating treated and untreated regions of hair in accordance with an embodiment of the present disclosure;FIG. 4 illustrates overlays shown on a display for AR glasses indicating regions of skin that are covered and uncovered in different gradations with a spray or lotion in accordance with an embodiment of the present disclosure;FIG. 5 illustrates overlays shown on a display for AR glasses indicating regions of a floor that are cleaned or uncleaned in accordance with an embodiment of the present disclosure;FIG. 6 illustrates an overlay shown on a display for AR glasses providing guidance for tool usage in accordance with an embodiment of the present disclosure;FIG. 7 illustrates an overlay and information panel shown on a display for AR glasses providing guidance for a welding task and related information in accordance with an embodiment of the present disclosure;FIG. 8 is a process diagram showing a spatial alignment technique in accordance with an embodiment of the present disclosure;FIG. 9 illustrates a sunscreen application system in accordance with an embodiment of the present disclosure;FIG. 10 illustrates an example AR predictive guidance system in accordance with an embodiment of the present disclosure;FIG. 11 illustrates a system for temporal tracking of materials in accordance with an embodiment of the present disclosure;4Attorney Docket No. 19842-006WOU1FIG. 12 shows an AR guidance system in accordance with another embodiment of the present disclosure;FIGS. 13A-13B illustrate a material-revealing illumination and coordinated active lighting control system in accordance with another embodiment of the present disclosure;FIGS. 14A-14B depicts a task-adaptive guidance through real-time skill-level and performance assessment system in accordance with another embodiment of the present disclosure;FIGS. 15-17 depict a system having shared autonomy with AR-coupled micro-actuation in accordance with another embodiment of the present disclosure; andFIG. 18 depicts a system for providing task-adaptive guidance through real-time monitoring of magnetic fields system in accordance with another embodiment of the present disclosure.DETAILED DESCRIPTION

[0012] The challenges that affect efficient use of products and tools in various settings arise from limited visibility of coverage, inadequate feedback during use, and lack of real-time or anticipatory guidance to guide users. The present disclosure addresses these issues by providing realtime visual feedback through augmented reality (AR) devices, such as glasses, based on data from connected sensors. Heatmaps, progress indicators, and guidance elements generated based on sensed data allow users to visualize consumption, adjust application amounts, and achieve improved outcomes, reducing waste and enhancing precision.

[0013] This is accomplished via systems comprising external sensors that may be integrated into various devices, tools, or packaging, and that are configured to wirelessly transmit data to AR devices that use the data to provide real-time information on the AR device display, including application-specific or general feedback, assisting with product usage and desired outcomes across multiple domains. In some embodiments, loT platforms may be integrated for enhanced connectivity, data analysis, and remote monitoring, making the system adaptable to evolving technologies.

[0014] These systems may include external sensors in an area where a task is to be completed and / or integrated into product packaging, wearables, tools, or industrial equipment, and are5Attorney Docket No. 19842-006WOU1positioned and configured to measure physical quantities such as liquid flow, surface coverage, temperature, application pressure, motion, or orientation. A communication module (preferably wireless) allows these sensors to transmit data to AR devices using protocols such as Bluetooth, WiFi, or similar. AR devices, including preferably glasses for many applications, are equipped with a processor, display, and receiver to receive data and provide real-time guidance to the user via overlays generated based on the data.

[0015] In some embodiments, an Al and computer vision module may be included that recognizes product orientation, shapes, labels, and task context of the user’s activity. This may include interpreting user intent and environmental conditions. An loT integration module connects the system to loT platforms for data storage, analytics, and remote monitoring, enabling functionality such as predictive maintenance and cross-device interaction. A user interface provides intuitive feedback via overlays, heatmaps, or visual instructions tailored to specific or general applications.

[0016] Exemplary applications are varied and may include any of the following. Personal care settings, in which sensors in bottle caps or applicators measure flow, orientation, and / or surface coverage, and AR glasses display a heatmap indicating treated and untreated areas or covered and uncovered areas. For cleaning tasks, spray bottles may have sensors that track surface coverage and orientation, and connected AR glasses guide users to missed spots for thorough cleaning. Product labels may be recognized and instructions or recommendations accessed to enable further guidance. In food preparation and cooking settings, sensors may be included in dispensers to monitor dispensed quantities of ingredients, such as oils and spices, and AR glasses display overlays for visualizing distribution patterns and recognize product types to help ensure recipe accuracy. For pharmaceutical packaging, dispensing monitoring may be used for dosage tracking to provide data that is used to generate AR visualizations of consumption and remaining quantities. For medical devices, sensors in medical instruments may measure dosage, pressure, or flow, and AR glasses provide visual feedback to assist with precision during procedures. In an agricultural setting, sensors in irrigation systems may measure water flow or coverage, and AR glasses guide users for optimal field management while optionally recognizing equipment types and labels.6Attorney Docket No. 19842-006WOU1

[0017] Tn addition, sensors in fitness wearables or medical patches may provide data for guided health information visualization. In the context of transportation and logistics, containers with sensors may track orientation and condition of cargo, and associated data is visualized through AR devices for logistics optimization. In industrial settings, sensors embedded in machinery may monitor operational parameters, and AR glasses provide information about system performance or identify issues in real time, including optionally recognizing specific machine parts or tools.

[0018] In addition, predictive guidance may be provided, such as suggesting the next ingredient in a cooking process or identifying areas most likely to require additional cleaning.

[0019] Further, virtual guidance can be displayed as overlays on floors for cleaning as viewed through AR glasses or on a display, showing where mopping or dusting has been insufficient, and guiding users in real time for optimal coverage.

[0020] In another context, visual feedback on consumption is provided in which the system highlights areas of under-application or insufficient product usage in real-time. For instance, AR glasses may display a heatmap or color-coded overlay to guide users on where additional product needs to be applied. This ensures even distribution and coverage for various applications, such as cleaning, personal care, or cooking.

[0021] In some embodiments, bottle caps for personal care products integrate flow sensors and communication modules to monitor product usage. AR glasses visually guide the user by indicating areas where the product has not been applied or insufficiently applied. Heatmaps or progress indicators can assist with even and thorough application of hair dyes, lotions, or creams.

[0022] Spray bottles for cleaning may include sensors to measure liquid usage and detect missed spots. AR glasses provide visual overlays to highlight untreated areas and guide users to adjust their application. Virtual markers on floors, walls, or other surfaces show areas needing additional treatment, dynamically adjusting based on real-time data.

[0023] Exemplary embodiments, applications, and techniques are discussed further in the sections that follow.Airborne Compounds7Attorney Docket No. 19842-006WOU1

[0024] Tn an embodiment, airborne compounds or gases, such as toxins, aromatics, scents, or aromas, are detected and information related to such detection is converted into overlays shown on AR devices, whereby the overlays provide information about the location, concentration, and / or type of compound or gas. Detecting, analyzing, and visualizing data for these compounds or gases may be performed using external aroma analyzers, image recognition, and AR glasses. The external aroma analyzer detects and measures volatile organic compounds (VOCs) and / or other gases, generating scent profiles. In some embodiments, Al processes interpret the detected data and provide visual overlays via AR glasses. Cameras, which may be integrated into the AR glasses or external devices, perform image recognition to identify objects such as fragrance bottles or other odor sources. The system enables measurable scent intensity feedback, color-coded mapping of fragrance components, Al-based recommendations, odor identification, and dynamic visualizations of scent diffusion. Applications may include fragrance selection, usage optimization, education, safety protocols, and odor management.

[0025] For airborne compounds, an external aroma analyzer detects VOCs and / or other gases, while a camera performs image analysis to identify objects or potential odor sources. An Al processor combines and interprets the data from the camera and the aroma analyzer, generating realtime overlays based in part on coordinating locations associated with the detected data that can be displayed on AR glasses. These overlays may include measurable scent intensity; diffusion or approximate spatial extent of detected aromas; color-coded mapping of detected scent components; object recognition for fragrance bottles or odor sources; feedback to prevent overuse or imbalance; and odor identification with actionable suggestions. In this way, the system is capable of quantifying scent intensity and diffusion, recognizing objects associated with odor sources, analyzing detected scent profiles, and presenting this information visually through AR glasses in a dynamic and user- friendly manner.

[0026] As shown in FIGS. 1-2, an aroma detection system 1100 includes AR glasses 1104 worn by a user 1102 and an aroma analyzer 1108 that is preferably external to the AR glasses. Aroma analyzer 1108 is configured to detect VOCs and / or other gases in the air and to measure scent intensity, composition, and diffusion, such as fragrant molecules emitted from a bottle 1112. A camera on or connected to AR glasses 1104 captures images of objects, scenes, or surfaces near the aroma analyzer 1108 or locations where the aroma analyzer has detected a compound of interest.8Attorney Docket No. 19842-006WOU1Object recognition is performed based on the captured images, such as for fragrance bottles, labels, or odor sources.

[0027] A processor 1108 analyzes and interprets aroma sensor data from the aroma analyzer and visual data from the camera and classifies scent components (e.g., floral, citrus, musky) using any suitable technique, such as machine learning. Actionable insights, recommendations, and visualizations may then be generated based on the classification and other data. This information is prepared as overlays 1120 (FIG. 2) to display in real time in the user’s field of view 1116 on AR glasses 1104. The overlays 1120 may include measurable scent intensity levels; color-coded scent mappings; visual representations of aroma diffusion or spatial influence; object recognition and metadata (e.g., perfume bottle details); and / or odor detection and mitigation recommendations. Such visual representations are illustrative and may reflect estimated or relative spatial characteristics rather than exact physical boundaries, particularly in environments affected by airflow, movement, or other environmental factors. Data collected by sensor 1108 includes spatial information that is translated to corresponding spatial points within the AR glasses display 1116 as described in more detail below.

[0028] As illustrated in FIG. 1, the aroma detection system 1100 comprises AR glasses 1104 worn by a user 1102, a discrete aroma analyzer 1110, and a central processor 1108. The aroma analyzer 1110 samples the environment to detect volatile organic compounds (VOCs), measuring properties such as intensity and composition from sources like a fragrance bottle 1112, and transmits this data (e.g., via wire or wireless communication) to the processor 1108. Simultaneously, a camera integrated into the AR glasses 1104 captures the visual scene. The processor 1108 fuses this olfactory and visual data to generate spatially registered overlays (shown in FIG. 2) within the user’s field of view 1116. These overlays dynamically visualize invisible scent characteristics, such as diffusion range or chemical notes, effectively mapping the detected aroma profile onto the physical world.

[0029] FIG. 2 depicts the system's operation from the user's perspective through the AR glasses frame 1104, which houses onboard electronics 1106. The user views the physical environment through the lens field of view 1116, seeing the fragrance bottle 1112 and the discrete aroma sensor 1110 (located on the table). Upon detecting a scent, the sensor transmits data via a wireless signal9Attorney Docket No. 19842-006WOU11114 to the processor 1108, which then calculates the aroma's spatial characteristics. In response, the system projects a virtual spatial overlay 1122 around the bottle. This overlay, rendered here as a volumetric envelope, is an illustrative representation of the aroma’s approximate diffusion range or intensity field, allowing the user to visualize the otherwise invisible extent of the scent molecules in real time.

[0030] Applications may include, for example, fragrance retail and selection, in which the system recognizes fragrance bottles and displays scent notes, intensity, and an approximate spatial representation of aroma diffusion, and uses color-coded overlays to visualize scent composition (e.g., citrus, floral, woody). In some embodiments, a fragrance blending module analyzes and balances scent components in real time. The system can provide measurable feedback to avoid fragrance over-application and visual warnings that highlight areas with excessive scent intensity. For odor detection and management, the system may identify unwanted odors and quantify their intensity using visual overlays that represent odor spread or relative influence and suggest mitigation actions (e.g., ventilation). Overlays may include a visual representation of scent diffusion or influence (e.g., 1120 in FIG. 2) that shows a dynamic range of detected aroma presence rather than a fixed physical boundary.

[0031] Further, gas leak detection may be enhanced by providing overlays showing gas leaks (e.g., methane, carbon monoxide) on AR glasses using integrated sensors. Real-time overlays can pinpoint the leak location and provide safety warnings to users, such as engineers, utility workers, or maintenance personnel. In emergency situations such as fires, chemical spills, or search-and-rescue missions, the AR glasses can identify dangerous fumes, toxic chemicals, or smoke intensity. The system can overlay hazard zones and guide responders to safer areas or individuals needing assistance.

[0032] In food industries, the system can detect spoilage or contamination by analyzing aroma profiles and overlay warnings for expired or unsafe items and assist quality inspectors in maintaining safety standards. In agricultural settings, the system can detect plant diseases, mold, or soil health indicators by analyzing crop or air emissions. The system can also be used to identify pollution sources, such as emissions or air quality changes. In a medical setting, the system can aid in detecting specific VOCs emitted from patients that may be associated with underlying health10Attorney Docket No. 19842-006WOU1conditions (e.g., diabetes, infections, or lung issues) (illustrative, non-diagnostic). In industrial or worksite settings, the system can identify dangerous chemicals or odors, providing visual overlays to detect hazards and guide clean-up processes.Hair Coloring

[0033] For hair coloring, a system as shown for example in FIG. 3 includes a camera module that may be located in various positions, including but not limited to, built into AR glasses 1204 for capturing images or reflections of the user’s head (e.g., via a mirror); integrated into a handheld haircoloring device to capture a direct view of the hair 1201; or embedded in a mirror in front of the user to provide an alternative angle and consistent imaging. The camera captures real-time images or video streams of the user’s hair for use in analysis and generation of visual overlays.

[0034] An AR system (e.g., glasses 1204, displays, or holograms) receives images from the camera and provides real-time overlays 1220 and visual feedback on a display 1216 of AR glasses 1204. The AR displays may include AR glasses with semi-transparent overlays; mirrors displaying virtual reflections of the user with augmented visuals; or holographic displays that project hair color visualizations in front of the user.

[0035] The data captured by the camera is processed to identify hair texture, condition, and color, and hair dye application is tracked in real time to identify treated and untreated areas of the user’s hair. The virtual overlays may include real-time hair color simulations (e.g., preview colors); highlighting untreated areas during the application process; and visualizing otherwise invisible dye application by displaying treated versus untreated sections, thereby assisting in achieving even application when the dye is initially difficult to perceive without the aid of the overlays. In some embodiments, the virtual overlays further include a translucent guidance region superimposed directly over the user’s hair to indicate a suggested next area for dye application, based on detected coverage patterns and application progress.

[0036] A device 1203 such as a hair dye applicator may include a communication module 1208 configured to send data to AR glasses 1204 and one or more detectors 1210 (e.g., conductivity sensors 1210a, 1210b) for detecting applied hair dye or other change in hair. Other sensors may be configured for detecting UV light imaging; thermal detection; spectral analysis; electrical11Attorney Docket No. 19842-006WOU1conductivity-based detection, which detects changes in electrical conductivity caused by the dye or developer; pH-sensitive detection, which detects pH changes on the hair surface due to the dye or developer; Optical Coherence Tomography (OCT), which captures high-resolution cross-sectional images to detect physical or chemical changes in the hair shaft caused by dye application; light polarization detection, which measures changes in surface reflectivity caused by dye application; acoustic or ultrasonic detection, which uses ultrasonic pulses to detect density or texture changes in treated hair; chemical markers or dye additives; and thermal conductivity detection, which measures heat absorption or retention differences between dyed and undyed hair using thermal sensors.

[0037] Data from sensors 1210 is processed in AR glasses 1204 and overlays 1220 are generated for providing guidance or other information on the display 1216, such as an area of hair that has been chemically treated. The system may include a hair coloring database of hair dyes, shades, and formulas that includes color profiles, fade timelines, and care recommendations that can be associated with detected products.

[0038] FIG. 3 illustrates an example hair coloring utility consistent with the embodiments described above. In this example, a user views their hair directly or via a reflection while wearing AR glasses 1204. A hair dye applicator 1203, such as a smart brush or applicator, includes a communication module 1208 and one or more sensors 1210 (for example, conductivity sensors 1210a, 1210b) configured to detect application of dye to hair 1201. Sensor data generated by the applicator 1203 is transmitted to the AR glasses 1204 and processed to generate a real-time overlay 1220 on an AR display 1216. The overlay visually differentiates treated and untreated hair regions, for example using color coding, cross-hatching, or other visual patterns, thereby assisting the user in achieving even coverage and identifying missed areas during the hair coloring process.Cleaning

[0039] Floor cleaning, or cleaning of other surfaces, whether through vacuuming, mopping, or dusting, requires precision and thoroughness to ensure cleanliness, especially in large or complex environments. Users often face challenges in tracking cleaned versus uncleaned areas, optimizing their cleaning path, and addressing hard-to-detect debris or surface contaminants during the cleaning process.12Attorney Docket No. 19842-006WOU1

[0040] AR glasses, when combined with detectors (e g., cameras, LiDAR, sensors, and image or signal processing techniques), can be used to provide information and guidance to a user while the user is cleaning by enabling spatial mapping, real-time overlays, and progress tracking. This guidance may be provided for hand-held or robotic cleaning tools and / or sensor-equipped manual tools to visualize cleaned versus uncleaned areas, optimize cleaning efficiency, and ensure thorough surface coverage based on detected surface conditions.

[0041] Spatial mapping may be provided using LiDAR, photogrammetry, 3D Gaussian Splatting, radiance field-based representations, and point-based or volumetric scene representations. Handle-based sensors on tools may be used for obtaining information about motion, orientation, and tool contact with a surface to be treated. Infrared detectors may be used for identifying wet or reflective surfaces. Fluorescent and UV detection may be used for fine dust or stains invisible under regular light. Thermal imaging may be used for temperature-based identification of surface contamination or spills. Ultrasonic mapping may be used to determine surface texture and surface transitions. Electromagnetic field (EMF) sensors may be used for detecting metallic debris. Electrostatic detection may be used for detecting charged dust particles. Optical flow mapping may be used for determining debris movement. RFID-based or QR-code tracking may be used for room positioning and coverage verification. Vibration feedback may be used to confirm tool engagement with surfaces. Camera-based detection may be used for real-time surface analysis and spatial mapping.

[0042] The guidance system may include the following: AR glasses equipped with cameras, sensors, and a visual display for providing real-time feedback via overlays that are spatially registered to and superimposed over physical surfaces being treated, such as translucent masks aligned to treated, untreated, or missed regions. The overlays may further include grids, heatmaps, color-coded zones, or other visual indicators that are co-registered with the physical surfaces or alternatively presented as auxiliary user interface elements, to convey coverage, progress, or guidance information.

[0043] Manual tools may include embedded sensors, which may include a camera module such as a depth camera and / or a color (RGB) camera mounted on a handle to analyze physical surfaces being treated and to support spatial registration of tool position relative to those surfaces. An inertial13Attorney Docket No. 19842-006WOU1measurement unit (IMU) may be used for tracking motion, position, orientation, and angle of the tool to enable alignment between tool movement and AR overlays superimposed on the physical environment. A LiDAR or depth sensor may be used for mapping a room or environment and identifying treated surfaces. Infrared sensors may be used for detecting reflective surfaces, wet areas, or streaks. UV or fluorescent light detection systems may be used to highlight dust, stains, or residues. Thermal sensors may be used for identifying temperature variations on a surface to locate spills or contamination. Electromagnetic sensors may be used for detecting metallic debris. Ultrasonic sensors may be used for mapping surface transitions and textures (e.g., carpet, tile). Air quality sensors may be used to monitor airborne particulates. Electrostatic sensors may be used for detecting fine, charged dust particles. Vibration sensors may be used to confirm physical engagement of the tool with the surface at locations corresponding to AR guidance overlays.

[0044] As shown in the example in FIG. 5, a robotic device 1401 may include cameras, LiDAR, ultrasonic sensors 1408, and / or other sensors or data sources suitable for localization, mapping, or navigation, for autonomous movement and surface treatment. Real-time treatment data may be synchronized with AR glasses 1404 that provide visual overlays 1420 (e.g., virtual trails) and 1421 (e.g., missed spots) spatially registered to surface 1405, showing progress, obstacles, or areas requiring manual intervention, as viewed on display 1416.

[0045] A guidance or strategy module processes input from AR glasses, manual tool sensors, robotic devices, and spatial mapping technologies to generate real-time maps indicating treated and untreated areas. Observations derived from AR-based sensing and spatial registration, including user interactions, coverage patterns, and environmental context, may be used to adapt or refine navigation paths, task sequencing, or execution behavior of manual tools or robotic devices. Patterns of user behavior, tool paths, and environmental surfaces may be analyzed over time to improve tool settings, robotic behavior, and treatment strategies for improved performance, and may further support system-level recommendations such as operational or maintenance alerts.

[0046] RFID tags or other location markers may be placed in rooms or environments to be treated to provide reference points that support spatial registration, coverage verification, or systematic progression through a treatment area.14Attorney Docket No. 19842-006WOU1

[0047] Tn an embodiment, a system includes a camera module mounted on the handle of a cleaning tool (e.g., mop, vacuum cleaner) to capture real-time images or videos of the floor. The camera is positioned and configured to capture high-resolution images or videos to identify cleaned and uncleaned areas. The video feeds are streamed to a processor for spatial analysis. This data is used along with data from LiDAR, and / or other sensors to map cleaning coverage accurately.

[0048] AR glasses are connected to the camera and sensors and display a real-time heatmap or grid overlay showing areas cleaned (e.g., green) and uncleaned (e.g., red). Alerts for missed spots or streaks may be shown.

[0049] In another embodiment, cameras are mounted both on the AR glasses and the tool handle. These cameras improve AR accuracy by providing real-time image data for the Al engine, enabling precise spatial mapping, better tracking of surfaces, and enhanced spatial registration of overlays relative to the physical environment. The tool handle camera may capture close-up details of the surface, such as dust, stains, or surface irregularities, while the AR glasses camera offers a broader spatial view to support comprehensive surface treatment coverage.

[0050] In another embodiment, the tool integrates a light-activated actuator system to dynamically adjust a tool head parameter, such as height, brush type, or suction setting, based on a detected surface type. The actuator may use minimal power and lightweight components for integration into manual or robotic tools and may adapt tool functionality for different surface types, such as carpets (e.g., increasing suction), hardwood floors (e.g., adjusting height), or tiles (e.g., brush adjustment).

[0051] In another embodiment, robotic surface-treatment tools, such as robotic vacuuming, mopping, or dusting tools, and other autonomous or semi-autonomous devices configured to treat surfaces, are equipped with LiDAR, cameras, and sensors for autonomous navigation. Real-time communication with AR glasses provides a visual representation of robotic operation progress and identifies areas requiring manual attention. The AR glasses display progress overlays showing treated versus untreated areas by the robotic device and alerts for areas that may require intervention (e.g., spills or regions not fully treated by the robotic device).15Attorney Docket No. 19842-006WOU1

[0052] Tn another embodiment, spatial mapping and navigation of a robotic device are dynamically influenced by information generated or refined through AR-based observation and user interaction. Visual data, corrections, or annotations captured by AR glasses during human-guided operation may be used to update or refine localization, mapping, or path-planning models of the robotic device, enabling subsequent autonomous operation to benefit from prior AR-assisted learning or intervention.

[0053] In another embodiment, spatial mapping, localization, and navigation of a robotic device may be dynamically influenced by information generated or refined through AR-based observation and user interaction. Visual data, spatial corrections, or user annotations captured by AR glasses during human-guided operation may be used to update, refine, or validate localization, mapping, or path-planning models of the robotic device. In this manner, subsequent autonomous operation of the robotic device may benefit from prior AR-assisted guidance, learning, or intervention.Tools and Applicators

[0054] Augmented reality (AR) glasses provide real-time visualization and feedback to users by generating spatially registered overlays superimposed on physical tools, hands, products, and application regions, based on depth imaging and optional machine learning. By capturing and analyzing the hand, arm, tool, or product and their spatial positioning relative to the environment, the system offers intuitive overlays that simulate interactions directly in the user’s field of view and ensure optimal application or operation.

[0055] Spatial information is captured using depth cameras and / or other spatial sensing modalities (e.g., LiDAR, stereo imaging, NeRF-based representations, 3D Gaussian Splatting (3DGS), RGB cameras, or similar) to determine and analyze the position, posture, shape, silhouette, and distance of a user’s hand and arm and a tool or applicator relative to the environment. Tools or products and their spatial orientation may be identified, and real-time AR overlays are provided that simulate interactions aligned with corresponding physical objects and display visual cues for optimal application or operation.

[0056] AR glasses may include one or more integrated depth cameras, or depth cameras may be external, to capture and synthesize spatial data in real time. A display system provides real-time16Attorney Docket No. 19842-006WOU1visual overlays spatially registered to tools, products, and application areas. A processing unit analyzes depth data and optionally Al-generated outputs to generate overlays. A machine learning engine or other suitable techniques may be used to provide tool and product identification based on the shape and size of the tool and the context in which the tool is being used. The 3D position, orientation, and interaction of tools and hands with the environment are determined.

[0057] A spatial mapping system includes depth sensors that capture spatial data for the hand, tool, and surrounding environment. A 3D representation of the environment, including objects, surfaces, and spatial relationships, may be generated, which may optionally be refined, augmented, or represented using techniques such as Neural Radiance Fields (NeRF), 3D Gaussian Splatting (3DGS), or other spatial reconstruction or representation methods.

[0058] A user interface includes visual overlays that are spatially registered to and superimposed over physical application regions and display relevant real-time contextual information for a live application, such as color-coded areas indicating applied and unapplied regions. A progress tracker may track the completeness of tasks such as sunscreen application, and interactive guidance may provide visual and textual instructions co-registered with corresponding physical locations for optimal use.

[0059] In this way, the system allows a user to visualize and be guided in the application of materials or processes that are otherwise invisible to the human eye, by presenting AR-based guidance, which may include spatially registered overlays superimposed on relevant physical surfaces, so that users can improve precision and thoroughness in tasks where visual confirmation is traditionally challenging.

[0060] As shown in FIG. 6, a tool 1501 such as a wrench includes a sensor 1508 (e.g., a force or torque sensor) configured to transmit real-time data to AR glasses 1504 worn by a user 1503. In this example, the wrench 1501 is used to tighten a nut 1507. The system captures image information via the AR glasses 1504, identifying the mechanical component (e.g., recognizing it as a specific fastener type or size, such as a 14mm nut). This visual data is combined with real-time torque or force data received from the sensor 1508.17Attorney Docket No. 19842-006WOU1

[0061] Based on this fused data, the system generates a dynamic overlay 1520 within the display 1516 (depicted here as a hatched arc gauge). This overlay provides continuously updated guidance, such as visually filling the arc as torque increases towards a target value, or displaying numerical readouts of the current force. This allows the user 1503 to precisely apply the correct torque while maintaining their gaze on the work piece.

[0062] In an embodiment for sunscreen application, as shown in FIG. 4, AR glasses 1304 may use imaging and / or depth sensing, which may include visible-light cameras, depth cameras, infrared sensors, optical flow analysis, inertial measurement units (IMUs), or combinations thereof, to capture a user's hand 1301 motion and the sunscreen bottle's 1303 orientation. Motion vectors derived from optical flow, optionally combined with orientation and movement data from one or more IMUs, may be used to estimate where and how the applicator has moved relative to the user's body over time, thereby inferring an application pattern without requiring direct detection or segmentation of the sunscreen on the skin. The application pattern may be analyzed and areas of the skin that are protected or unprotected may be identified. The AR display provides a real-time overlay 1320 that distinguishes protected areas (e.g., darkened zones) from unprotected areas (e g., clear), and which may be spatially registered to the user's skin, on display 1316.

[0063] In other embodiments, users can be guided with the application of insect repellent, skin cream, or similar, in which the system identifies the product and its intended application area using visual characteristics, contextual cues, and / or sensed motion and orientation, which may include depth imaging and optional Al-based classification. Visual overlays guide the user in applying the product evenly and comprehensively, and progress trackers indicate the percentage of the skin covered. The coverage may be inferred over time based on observed applicator movement relative to the user’s body rather than direct sensing of the applied material.

[0064] In other embodiments, tool guidance is provided for hand-held devices in which sensors capture the 3D position and orientation of hand-held tools (e.g., paintbrushes, spray applicators), which may be derived from imaging, depth sensing, optical flow, IMUs, or combinations thereof, and the system displays visual overlays via AR glasses to guide the user in achieving precise application or operation. The overlays may represent target paths, coverage regions, alignment cues,18Attorney Docket No. 19842-006WOU1or deviation indicators, and Al may adapt the overlays based on the tool’s detected position, orientation, and motion over time and the task’s requirements.

[0065] In operation, upon activation the AR glasses may initialize one or more imaging and / or depth-sensing components to observe the immediate environment and objects within the environment. The glasses may establish a 3D representation of the surroundings, capturing objects, surfaces, and spatial relationships, or may operate using partial or task-specific spatial context without constructing a complete 3D map. The user selects the task type (e.g., applying sunscreen, using a hand-held tool) via a built-in or paired user interface, which conditions how sensor data is interpreted and how guidance overlays are generated.

[0066] The depth cameras and / or other imaging sensors capture the user’s hand, arm, and the tool or product being used. The system identifies the tool or product based on its shape, texture, and other contextual information, if available, such as tags (e.g., RFID, QR codes, or similar). Positional data such as orientation, posture, and distance relative to the user and environment is processed in real time. This positional data may be derived from image-based tracking, optical flow, inertial sensing, or combinations thereof. The spatial mapping system may integrate the captured data to create a detailed 3D representation of the scene, including the user’s hand, tool, and nearby objects or surfaces. Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) or similar may be employed alongside depth cameras (e.g., LiDAR, photogrammetry) to enhance 3D reconstruction accuracy. For invisible material applications (e.g., sunscreen, window cleaning fluid), the system identifies the target area and overlays guidance for application precision, wherein such guidance may be generated based on inferred interaction between the tool or applicator and the target area over time rather than direct sensing of the applied material.

[0067] If, for example, the object held is a wipe or sponge, the system may use pre-trained information specific to the product, such as expected contact footprint, deformation behavior, or typical motion patterns, to predict how the wipe or sponge should move and operate. By tracking the shape and silhouette of the object at successive frames as the hand moves, and optionally combining this with motion vectors and orientation data, the system estimates the surface area contacted over time and calculates the total area classified as “cleaned.” The system can then provide an overlay with that information on the AR display in real time.19Attorney Docket No. 19842-006WOU1

[0068] The real-time overlays are generated for the AR glasses. For colorless materials, protected areas are highlighted (e.g., in a first color) in the overlay, while uncovered areas remain neutral or are highlighted in a different color. Another overlay may show the tool’s trajectory and points of contact, assisting the user in precise placement. Spatial indicators, such as distances, alignment lines, or zones requiring attention, may also be displayed. The overlays may be view- locked, object-anchored, or spatially registered to physical surfaces, depending on the task and available sensing. Feedback is dynamically updated based on the tool’s movement or application progress. Recommendations may be provided to the user via the AR glasses based on system learning of the user’s behavior or object’s behavior and the context of the task, including predictive guidance that anticipates likely missed areas or upcoming deviations based on observed motion patterns. For example, if uneven sunscreen application is detected, the system suggests additional coverage based on inferred coverage gaps from motion history. If the tool deviates from an intended path, corrective visual cues are provided.

[0069] The system may track progress using visual markers and progress bars, such as coverage percentage for tasks like sunscreen or cream application or precision metrics for industrial or recreational applications, such as hole size matching for a drill or alignment for a wrench, wherein such metrics may be derived from accumulated motion history, orientation data, and inferred interaction between the tool or applicator and the target area over time.

[0070] Upon task completion, the system may verify and validate the results. In colorless materials applications, all target areas are checked for sufficient coverage, which may include confirming that inferred coverage metrics exceed a threshold or satisfy task-specific criteria. In industrial tasks, measurements are confirmed to match specified dimensions or alignments, based on sensed tool position, orientation, and spatial relationships relative to reference features.

[0071] A depth camera and / or other imaging sensors may capture spatial data about the environment, including objects, surfaces, the user’s hand, and tools, and provide raw spatial information such as distances, angles, and relative positions. AR glasses may capture additional visual data or stream it from other cameras (if available) and provide user input such as gestures, movements, or gaze tracking, which may be used to condition or refine guidance overlays and user interaction.20Attorney Docket No. 19842-006WOU1

[0072] Tn some embodiments, the system may perform object recognition to identify a tool being used (e.g., a spray bottle or a drill) and to determine its immediate operating context (e.g., a surface being treated or a material being drilled). Based on the identified tool and context, the system may further perform scene understanding by interpreting spatial relationships between the tool, the user’s hand, and nearby surfaces or boundaries relevant to the task. This scene understanding may be derived from processing spatial data to determine where interactions are occurring and where guidance should be provided. In some embodiments, scene understanding includes creating a 3D map of the environment, while in other embodiments it includes establishing a partial or taskspecific spatial model sufficient to support guidance and feedback during the operation. Spatial mapping may combine depth data and visual input when such mapping improves guidance accuracy for the selected task.

[0073] AR overlay generation in AR glasses includes projecting augmented overlays, such as visual guides including alignment markers, measurements, or edges of application areas (e.g., green for protected skin, red for unprotected areas). Overlays may also include interactive elements such as progress bars or alignment corrections for enhanced guidance, and may be updated reactively or predictively based on inferred progress, anticipated deviations, or upcoming task steps.

[0074] Users receive visual feedback and make adjustments based on the overlays. Interactive features, such as gesture controls or voice commands, may allow users to toggle settings, adjust sensitivity, or refine overlays. The system may adapt overlays dynamically as the user works, providing continuous guidance based on observed user interaction, tool movement, and inferred task progress.

[0075] Task completion, such as full sunscreen coverage, accurate tool alignment, or complete cleaning, may be confirmed via an indicator provided via an overlay, wherein such confirmation may be based on inferred completion criteria derived from motion history, spatial alignment, or taskspecific thresholds.

[0076] In an example, a depth camera and / or other imaging sensors capture spatial data of a user holding a drill. An Al processing unit identifies the tool and recognizes the task as drilling based on observed geometry, motion patterns, or contextual cues. Spatial mapping is used to determine the drill’s position and estimate relevant dimensions of the workpiece. An AR overlay 21Attorney Docket No. 19842-006WOU1displays the alignment of the drill with the workpiece, showing target points for drilling and depth indicators. User interaction refines the alignment, and the system dynamically updates guidance in real time as the drill position and orientation change.

[0077] Sensor data is used to recognize or infer material properties such as texture, absorption behavior, or flow characteristics based on observed interaction between the material, applicator, and target surface. For instance, the system differentiates between water-based and oil-based materials (e.g., sunscreens, creams, or adhesives) by analyzing how the material spreads, adheres, or accumulates during application, and tailors visualization overlays accordingly. In addition, multiple layers of applied materials are tracked over time, such that uneven reapplication can be highlighted or a need for additional coating is indicated based on inferred layering or coverage patterns.

[0078] Sensed environmental data, such as wind speed, humidity, temperature, or lighting conditions, is used to adjust application recommendations. For example, windy conditions can cause spray products to deviate from intended trajectories when applied using assumptions appropriate for calm conditions. In response, the system adjusts overlay guidance to compensate for environmental effects by modifying suggested aim, sweep patterns, or coverage zones to improve application accuracy under current conditions.

[0079] Additionally, haptic feedback is provided in some embodiments through wearable devices such as gloves, wristbands, or other body -worn interfaces to complement AR visual guidance. For example, tactile cues such as vibration, pressure, or force feedback, optionally combined with audio cues, indicate when a tool deviates from a prescribed path, approaches a boundary, or requires corrective action.

[0080] In another embodiment, the system is applied to welding operations, where rapid changes in optical intensity, heat, and material reflectance create safety and quality-control challenges. During welding, conventional auto-darkening helmets rely on local optical sensors that transition to a darkened state only after detecting an increase in luminance from a welding arc, and even short reaction delays can expose the user to harmful radiant energy. The systems described herein improve safety and precision by using external sensors, spatial mapping, and predictive processing, including predictive Al in some embodiments, to anticipate the location and timing of a22Attorney Docket No. 19842-006WOU1welding arc before ignition and to provide real-time welding-quality visualization and guidance during the operation.

[0081] One or more sensors positioned on or integrated into a welding tool, helmet, glove, or nearby fixture detect physical quantities indicative of imminent arc formation or ongoing weld conditions. These sensors detect parameters such as luminance, thermal signatures, motion or orientation of the welding tool, electrode proximity, or reflectance changes on the workpiece. The sensors transmit the detected values and associated spatial parameters to the AR glasses. Using the spatial parameters describing tool pose, relative position to a weld joint, and observed motion trends, the system infers an expected arc location and timing within a virtual representation of the workspace, enabling pre-emptive visual or haptic feedback prior to arc ignition and continuous guidance during welding.

[0082] Using the coordinate-frame alignment techniques described below, the AR glasses convert the sensor-reported spatial parameters into the AR world frame and generate overlays that may be anchored to a predicted weld location. The system may pre-darken or partially darken the display in anticipation of a welding arc based on predicted tool position or imminent ignition, thereby reducing delay-based exposure relative to conventional auto-darkening helmets that react only after arc detection.

[0083] As welding proceeds, additional overlays may be generated based on sensed optical intensity, thermal gradients, reflectance characteristics of the molten pool, and motion of the welding tool. The system may display weld-quality indicators, such as bead uniformity, travel speed consistency, or estimated penetration depth, derived from combined sensor data and task-specific processing, including Al-based analysis in some embodiments, wherein variations in optical intensity, molten pool geometry, thermal distribution, and tool motion are correlated with weld formation characteristics over time. Overlays may highlight deviations, insufficient coverage, overheating, or inconsistent travel speed along the joint. When spatial anchoring is available, the overlays are aligned with the physical weld seam so that the user receives real-time guidance directly in the AR field of view.

[0084] In another embodiment suitable for welding operations, the system addresses safety and quality control challenges. As shown in FIG. 7, a welder helmet 1604 (which may comprise a23Attorney Docket No. 19842-006WOU1protective mask with an integrated AR display 1616, or accommodate AR glasses worn underneath) provides protection and visual guidance. A camera (e.g., an external scene camera on the helmet or glasses) captures the welding tool 1601 and the workpiece environment.

[0085] The tool 1601 includes a sensor and data module 1608 (detecting parameters such as tool orientation, thermal signatures, or arc stability) in communication with the helmet system. Data from the sensor module 1608 and the camera is processed to generate a guidance overlay 1620 (e.g., indicating predicted weld path or puddle characteristics) to act as a real-time guide. Additionally, a specific data window 1621 may be displayed within the AR view 1616, presenting critical metrics such as arc stability, travel speed, or remaining wire feed, derived from the sensed context.Spatial Alignment

[0086] In certain embodiments, measurements from external sensors are coordinated with the AR view by placing both into a common three-dimensional coordinate system. In such embodiments, the AR glasses establish a three-dimensional coordinate system representing the physical environment using depth sensing, SLAM-based tracking, LiDAR, stereoscopic cameras, Neural Radiance Fields, 3D Gaussian Splatting, or combinations thereof. This coordinate system provides a spatial reference in which surfaces, objects, or regions relevant to a task are associated with stable spatial coordinates over time, enabling guidance overlays to persist, update, or reappear consistently as the user moves. Each external sensor reports not only the detected physical value but also spatial parameters such as position, orientation, distance, or coverage footprint. Using these spatial parameters, the system determines a geometric transformation between a coordinate frame of the external sensor and a coordinate frame of the AR glasses, allowing sensor detections to be converted into the AR world frame and rendered as overlays at corresponding physical locations, such that even when the sensor has a different viewpoint than the AR glasses, the overlay appears at the correct physical location in the user’s field of view.

[0087] When an external sensor detects a physical quantity at a particular location, the sensor additionally provides one or more spatial parameters describing the detection. These spatial parameters may include a position of the sensor, an orientation of the sensor, a distance from the sensor to the detected region, a footprint or coverage region associated with the detection, or a time- stamped pose of the tool or device in which the sensor is embedded. These spatial parameters enable 24Attorney Docket No. 19842-006WOU1the system to associate the detected physical quantity with a specific region or interaction point within the task environment.

[0088] In some embodiments, using the spatial parameters provided by an external sensor together with the spatial context established by the AR glasses, the system determines how a detected physical quantity relates to objects, surfaces, or regions within the environment. This determination may include evaluating where the detection occurred relative to task-relevant features, how the detected quantity evolves over time, and whether the detection corresponds to an intended interaction area or an unintended deviation from a desired operation.

[0089] To align detections from the external sensor with the AR glasses’ view, the system determines a geometric transformation between a coordinate frame of the external sensor and the coordinate frame of the AR glasses. This transformation may be obtained using one or more of the following techniques: visual localization of the sensor by the AR camera; fiducial markers or machine-readable labels on the sensor or tool; inertial-measurement comparisons; known mounting geometry; radio-frequency ranging; depth-based proximity estimation; or simultaneous mapping of the shared environment by both devices.

[0090] Using the transformation, the processor converts the location associated with the physical quantity detected by the sensor into the coordinate frame of the AR glasses. The system then renders an overlay (e.g., heatmap region, indicator, highlight, or guidance marker) at the corresponding location within the virtual representation of the environment. The conversion accounts for relative pose, distance, and orientation differences between the sensor and the AR glasses so that the rendered overlay corresponds to the same physical region detected by the sensor. As the user moves, the overlay remains anchored to the correct physical position, updating based on changes in user viewpoint while maintaining spatial correspondence, ensuring that information detected by the external sensor is visually aligned with the user’s real-world view of the same region.

[0091] This allows the system to coordinate spatial measurements obtained from a sensor having a different point of view than the AR glasses, and ensures that overlays displayed in the AR interface correspond accurately to the actual physical locations at which the sensor detected the25Attorney Docket No. 19842-006WOU1physical quantity, even when the sensor and the AR glasses observe the environment from different perspectives or at different distances.

[0092] For example, the AR glasses can generate a three-dimensional coordinate system of the environment in which the task is being performed using SLAM (simultaneous localization and mapping), depth cameras, LiDAR, or three-dimensional reconstruction techniques such as Neural Radiance Fields or Gaussian Splatting. The three-dimensional map gives each point in the environment an (x, y, z) coordinate in the AR glasses’ world frame, providing a consistent spatial reference for associating physical surfaces, objects, or regions with corresponding overlay locations rendered through the AR display.

[0093] The data from the one or more external sensors also includes spatial parameters related to measurements made by the sensors for each detected physical quantity, such as material presence or coverage (e.g., sunscreen), chemical concentration (e.g., volatile organic compounds or VOC), or optical characteristics (e.g., surface reflectance). The external sensor data may include the sensor’s pose or orientation, a direction of detection, a relative distance between the sensor and a location of detection, and / or a region-of-interest footprint associated with a tool motion or sweep path. These parameters are expressed in the sensor’s coordinate frame and are converted into the AR glasses’ world frame using the coordinate-frame transformation described herein.

[0094] A coordinate-frame transformation aligns the sensor’s coordinate frame with the AR glasses’ world frame. In some embodiments, the system computes a rigid-body transformation that maps points measured by the sensor into the AR glasses’ coordinate system. This transformation may be estimated using one or more of the following: fiducial markers (e.g., QR codes, AprilTags); inertial measurement units on the AR glasses and the external sensor; visual alignment when the sensor is within the field of view of the AR glasses; known tool geometry; wireless ranging constraints (e.g., Bluetooth or ultra- wideband); or known mounting geometry, such as when the sensor is integrated into or rigidly attached to a device including a mop, bottle, or applicator.

[0095] Once this transformation is established, sensor-detected location information is mapped into the three-dimensional world frame of the AR glasses, and an overlay is rendered at the AR- world coordinate corresponding to the sensor’s detection. This mapping allows physical conditions sensed by the external sensor to be visualized directly at their associated real-world locations 26Attorney Docket No. 19842-006WOU1through the AR display. For example, if a sensor on a mop detects a missed region based on relatively lower reflectance values compared to surrounding areas at a position (x', y', z') in the mop’s coordinate frame, that position is transformed into the AR glasses’ coordinate frame according to:(x, y, z) = T_{{AR} -{ sensor}} (x', y', z')

[0096] An overlay is provided on the AR glasses display at those coordinates (x, y, z) that correspond with the sensor measurement expressed in the sensor’s coordinate frame, after transformation into the AR glasses’ world frame, so that the overlay visually aligns with the physical location associated with the detected physical quantity.

[0097] As outlined in FIG. 8, cameras associated with the AR glasses in system 800 capture data about a spatial region to define an AR view coordinate frame 804. Simultaneously, the external sensor system 802 detects physical quantities and defines a sensor coordinate frame 806, associating a specific location within that frame with each measurement. These distinct spatial coordinate frames are aligned via a geometric transformation 810 that represents the spatial relationship between the external sensor and the AR glasses. This transformation is determined via AI / ML inference and sensor fusion 812 based on one or more parameters, such as visual localization, inertial-measurement comparison, wireless ranging, known geometry, or simultaneous mapping of a shared environment. The resulting coordinate transformation is then applied to convert measurements from one frame of reference to another 820, thereby allowing an AR overlay based on sensor data to be rendered on the camera view 830 in precise alignment with the real-world environment.Overlay Guidance

[0098] Overlays that provide information and guidance to users may include visuals that depict treated or untreated areas, cleaned or uncleaned areas, or similar conditions of a surface. This may be accomplished through various analysis techniques, including analysis of captured light data. In one embodiment, live red-green-blue (RGB) video frames are received from a camera positioned to27Attorney Docket No. 19842-006WOU1capture the pertinent environment and converted to a hue-saturation-value (HSV) color space to reduce sensitivity to illumination variation. Region segmentation is performed using image-based segmentation techniques, which may include a pre-trained convolutional neural network, rule-based thresholding, edge clustering, or combinations thereof. For each segmented region, temporal changes in texture or appearance are evaluated over successive frames, such as over a time interval of approximately 1.5-3 seconds.

[0099] A region is determined to have received sufficient interaction based on analysis of motion vectors derived from optical-flow computations across successive image frames. Optical- flow techniques are applied to the video to estimate per-pixel or per-region motion vectors that indicate movement of a tool or applicator relative to a surface. The system evaluates both the spatial distribution and temporal persistence of these motion vectors to infer whether sufficient interaction with the region has occurred. A region may be designated as having received adequate treatment if (i) the density and duration of detected motion over the region exceed a threshold indicative of sufficient activity, and / or (ii) a correlated change in surface appearance is detected, such as a reduction in specular highlights or coloration consistent with material application, material removal, or surface modification. In this manner, the same analysis framework may be applied to tasks including cleaning, coating, painting, polishing, or application of lotions, creams, or similar materials.

[0100] In another embodiment, the system includes a trained machine learning model that analyzes motion vectors generated by optical-flow computations in conjunction with visual cues, such as color changes or specular highlights, to determine whether a given region has received sufficient interaction or treatment. The model may be trained on a labeled dataset comprising video sequences of a given activity, annotated with labels indicating effective and ineffective interaction outcomes, which may include cleaning, coating, material application, surface modification, or similar operations. Features input to the model may include average device velocity over time in a region, stroke or motion pattern vectors, changes in surface specularity, and delta-color histograms in a color space such as HSV. The trained model outputs a probability or confidence score indicating interaction or treatment effectiveness, which may be used to classify regions along a continuum or thresholder into states such as treated / untreated or sufficient / insuffi cient, enabling robust determinations under varying lighting conditions, user styles, and surface geometries.28Attorney Docket No. 19842-006WOU1

[0101] From the regions determined to be treated or untreated, or assigned confidence values or gradations thereof, a mask representing interaction or treatment state is generated and encoded in an overlay map, which is transmitted to the AR glasses or another display device for visualization. The mask may represent binary states, confidence-weighted values, or continuous gradients corresponding to degrees of completion or coverage.

[0102] Using the masks obtained as described above, region-level labels and corresponding spatial extents may be derived for one or more regions of interest, optionally combining automatically generated proposals with manual or semi-automatic refinement. In some embodiments, these labeled regions are used to train a detection model capable of identifying similar regions in real time on resource-constrained hardware. The detection model may be a single-stage neural -network detector or another lightweight classifier, and real-time operation may involve receiving image frames from a camera and producing spatial prompts, bounding regions, or labels that are provided to a segmentation or overlay-generation module.

[0103] In some embodiments, the system implements a unified multi-sensor fusion framework in which the AR device establishes the primary spatial reference frame using simultaneous localization and mapping, depth sensing, inertial measurements, and, in certain embodiments, scenereconstruction techniques such as photogrammetry, neural radiance fields or Gaussian-splatting models. These processes generate a stable three-dimensional representation of the user’s environment that serves as the world coordinate frame. External sensors, including cameras, IMUs, force or torque transducers, flow or dispensing sensors, thermal or UV detectors, airborne-compound analyzers, ultrasonic sensors, LiDAR modules, and other sensing modalities described herein, are each associated with a known or estimable calibration relationship that relates the sensor’s local coordinate system to the geometry of the tool, package, or object in which the sensor is integrated. As the AR device observes such tools or objects through fiducial tracking, object recognition, depthbased pose estimation, structured-light markers, or inertial correlation, the system computes a realtime or periodically updated transform from each sensor into the AR world frame, thereby enabling each sensor’s measurements to be spatially registered to real -world surfaces, regions, and structures.Example 1 : Application of Lotion or Similar29Attorney Docket No. 19842-006WOU1

[0104] Environmental sensors positioned within the room or incorporated into appliances, fixtures, or robotic platforms may also provide measurements that are transformed into the AR coordinate frame. These environmental sensors may either have a known fixed installation geometry or may report a dynamically determined pose obtained through their own inertial units, wireless ranging systems, or camera-based localization. Measurements from these external devices, which may include thermal readings, UV intensity, airborne-compound concentration, distance or depth information, or other environmental parameters, are therefore integrated into the same world- referenced model used by the AR glasses. In this manner, sensing information originating from heterogeneous devices, with differing modalities, fields of view, and operating frequencies, can be combined within a single spatial representation.

[0105] A temporal fusion module, which may employ filtering techniques such as a Kalman filter, extended Kalman filter, or particle filter, integrates the asynchronous and multi-rate data streams produced by these various sensors. High-frequency inertial signals, mid-frequency camera and depth measurements, and lower-frequency physical, chemical, or environmental readings are combined to generate a temporally coherent estimate or confidence- weighted estimate of tool orientation, surface state, material deposition, environmental field intensity, and other sensed physical quantities. By maintaining all sensor inputs within a unified coordinate framework, the system enables the generation of AR overlays, such as coverage heatmaps, gradients of material density, predicted flow or dispersion patterns, hazard or contamination zones, and directional guidance cues, that remain spatially and temporally aligned with real -world objects. This fused representation further enables robust performance in the presence of occlusions, sensor noise, or partial visibility of tools or surfaces, and allows the system to synthesize information that likely no single sensor could reliably produce on its own.

[0106] In one illustrative example, as shown in FIG. 9, the system assists sunscreen application by using a data-driven inference engine that processes data from multiple sensors to infer product deposition and coverage quality. The sunscreen container provides dispensing information, such as flow rate or quantity dispensed over time, while the user’s hand or applicator provides motion and orientation data through inertial sensing. The AR glasses capture RGB, depth, and ultraviolet imagery of the user’s skin, and these data streams are aligned to the AR world coordinate frame using the tracking methods described herein. The data-driven inference engine which in some 30Attorney Docket No. 19842-006WOU1embodiments is a machine-learning engine receives the fused, spatially aligned sensor data and analyzes them collectively to estimate where sunscreen has been applied, how evenly it has been distributed, and whether additional product is needed in untreated regions.

[0107] The model may evaluate visual cues such as reflectance changes under UV illumination, combine them with known dispensing amounts and motion patterns, and identify inconsistencies that would not be detectable using a single sensor alone. For example, the Al engine may determine that although product was dispensed, the corresponding region of skin shows insufficient change under UV imaging, indicating that the product did not reach the intended area. Conversely, the engine may recognize surface changes consistent with sunscreen absorption even when dispensing data is low, thereby distinguishing between actual material deposition and sensor noise or lighting artifacts. By learning correlations between these different modalities, the system produces a more accurate realtime coverage map and a more reliable estimate of total product consumption.

[0108] FIG. 9 depicts the system in operation, where a user 10 applies a substance to a target surface 12 (e.g., skin) using a smart applicator device 14 while wearing AR glasses 16. The glasses feature a headset sensor suite 32 that captures the environment. The field of view (FOV) 18, indicated by dashed lines extending from the user’s eyes, delimits the area where digital augmentations are visible. Within this FOV, the system renders a virtual overlay region 20 directly on the target surface 12. This overlay provides spatially registered feedback, visually distinguishing between a "covered" zone 22 (e.g., dense cross-hatching) where product has been successfully applied, and a "missed" zone 24 (e.g., open pattern) requiring treatment. As the user sweeps the device in a direction of motion 26, the applicator performs an active sensor / dispensing action 28, simultaneously transmitting telemetry via wireless communication signals 30 to the headset. To further assist the user, a virtual floating UI 34 displaying "Live Coverage Data" is projected in the peripheral view, remaining visible only through the AR glasses.Example 2: Predictive Task Guidance31Attorney Docket No. 19842-006WOU1

[0109] Tn another embodiment, the system generates predictive task guidance by analyzing user motion, tool trajectory, environmental context, and sensor feedback to infer an intended action before that action is completed. The AR device tracks the user’s hand, head, or tool movement using visual tracking, depth estimation, inertial sensing, or any combination thereof. These signals are processed together with external sensor measurements, such as dispensing flow rate, torque feedback, thermal output, surface reflectance, or airborne-compound detection, to produce a temporal sequence characterizing the evolution of the user’s current behavior. The processor evaluates changes in motion direction, velocity, orientation, and sensor values over time to form a short-horizon behavioral trajectory. By comparing the current trajectory and sensor observations to previously observed patterns stored in the system’s data-driven inference engine (including a machine-learning engine in some embodiments), the processor generates a prediction of the user’s likely next action, target region, or intended application.

[0110] Based on this predicted intent, the system may generate task guidance overlays prior to the user completing the associated movement. For example, the system may display a projected path of a tool, a recommended region of application, an anticipated material deposition zone, or a caution indicator corresponding to an undesirable outcome such as excessive pressure, misalignment, or over-application of product. These overlays are spatially aligned with the physical environment and represent an expected future interaction state rather than a measured past condition. The predictive overlays may dynamically update as the user changes speed, direction, or behavior, allowing guidance to be provided proactively rather than only in response to errors that have already occurred.[oni] In some embodiments, predictive modeling incorporates environmental cues including surface geometry, illumination conditions, material properties, or environmental fields measured by thermal, UV, or chemical sensors. The processor analyzes these contextual factors together with user-motion data to estimate whether the predicted trajectory will intersect, contact, or sufficiently cover an intended target region and whether the predicted interaction is likely to achieve the required coverage, alignment, or dosage. If the system predicts insufficient material distribution, improper contact angle, or deviation from a desired path, the AR device may provide early corrective indications, such as directional arrows, contour highlights, or suggested adjustments, before the predicted outcome occurs, assisting the user prior to manifestation of an undesirable result.32Attorney Docket No. 19842-006WOU1

[0112] As illustrated in FIG. 10, the guidance system assists a user 200 (e.g., a stylist) treating a subject 206. The user wears AR glasses 202 and manipulates an applicator tool 205 (such as a brush) with their hand 204. The system tracks the tool’s spatial trajectory, establishing a specific movement vector 210 relative to the subject, while the glasses maintain a field of view 212 for continuous sensor tracking. Wireless data links 214 facilitate real-time communication between the tool, the glasses, and an intent modeling processor 220. This processor analyzes the kinematic data to forecast the user's intended stroke, by projecting a predictive guidance region 230, such as a virtual overlay indicating the optimal target area, onto the subject via the AR display. Simultaneously, the system updates a completed / matching region 232 to visually distinguish areas where the application has already been successfully performed.Example 3: Temporal Tracking

[0113] In another embodiment, the system performs temporal tracking of materials that are invisible, semi-visible, or intermittently visible to the user or to conventional imaging sensors. Many consumer and industrial materials, such as lotions, gels, soaps, chemical cleaners, disinfectants, adhesives, and certain cosmetic formulations, exhibit appearance changes over time due to absorption, evaporation, diffusion, polymerization, or surface interaction. The system captures a time series of sensor measurements from the AR device and any external sensors, including RGB imaging, depth data, ultraviolet or infrared reflectance, thermal signatures, moisture sensing, airborne-compound detection, or flow information from a dispensing container. Each measurement in the time series is associated with a spatial location on the surface within the AR world frame, and by aligning these measurements within the unified spatial coordinate framework described herein, the system generates a temporally indexed record of material interaction with the surface.

[0114] The processor analyzes these temporally aligned measurements to estimate dynamic material properties such as diffusion radius, absorption rate, surface dwell time, drying rate, or chemical reaction progression. In some embodiments, the system detects subtle changes in reflectance, sheen, fluorescence, or surface texture that occur gradually and may not be perceptible in a single frame. By comparing measurements from successive time points at the same spatial location, the system evaluates the progression of these changes over time and infers the presence,33Attorney Docket No. 19842-006WOU1concentration, movement, or transformation state of a material that is not reliably detectable in any single snapshot. In this way, the system uses temporal patterns to characterize the material state.

[0115] The system may further predict future material behavior by extrapolating the observed temporal evolution. For example, the processor may estimate when a surface will reach adequate absorption, when a cleaner will complete its dwell time, or when a semi-transparent coating has reached uniform spread. These predictions are derived from observed rates of change and previously learned or modeled material behaviors. Predictive overlays may indicate regions at risk of non- uniform drying, insufficient coverage, or delayed reaction completion. The overlays are spatially registered to the affected surface regions and represent an expected future material state, allowing the user to adjust application technique, redistribute product, or wait for a required time threshold before proceeding to the next step of a process.

[0116] In one illustrative example, the system assists with applying a transparent skincare serum. The serum appears glossy at first but becomes nearly invisible as it absorbs. The AR device detects the evolving reflectance pattern under visible and UV light, while a packaging-integrated sensor reports the amount dispensed. By comparing sequential frames captured over time and correlating them with the dispensed quantity, the system evaluates the rate and uniformity of absorption across different skin regions and determines which regions are still wet, which are partially absorbed, and which require additional product. The system may also identify streaking caused by uneven distribution by analyzing the temporal gradient of surface reflectance. Through this time-aware approach, the system provides actionable feedback that improves application quality and reduces product waste.

[0117] FIG. 11 illustrates the predictive outcome process through chronological frames. At Time T1 (Application), a dispenser tool 302 deposits a fresh material layer 304 onto the target surface 300 (e.g., skin epidermis). The system employs sensor polling paths 305 to analyze the surface reflectance, generating a detected wet region 312 visualized within an AR overlay box 310. At Time T2 (Mid-Absorption), as the substance transitions to an absorbing material layer 304', a temporal comparator processor 320 analyzes the rate of change between T1 and T2 data. This analysis triggers an AR overlay box 322 that highlights absorption anomalies like slow diffusion. Finally, at Time T3 (Predictive Outcome), the system projects a predictive coverage quality map34Attorney Docket No. 19842-006WOU1330, delineating a sufficient coverage region 332 where absorption is confirmed, while simultaneously identifying a coverage risk region 334 where dwell time is predicted to be insufficient, prompting the user for corrective action.Example 4: Adaptive Visualizations

[0118] In another embodiment, the system generates adaptive AR visualizations whose appearance and content change dynamically based on real-time sensor data, environmental conditions, or inferred task state. The processor determines the reliability, confidence level, or relevance of each sensing modality, such as RGB imaging, depth sensing, UV illumination, thermal detection, flow-rate measurements, chemical sensing, or inertial tracking, and adjusts the displayed overlays accordingly. The reliability determination may be based on signal stability, noise level, occlusion state, consistency over time, or agreement with other sensing modalities. When a particular modality exhibits higher certainty or clearer discriminative power for the current task, the system may emphasize overlays derived from that modality by increasing their opacity, contrast, spatial prominence, or temporal update rate. Conversely, if a sensor becomes unreliable due to occlusion, motion blur, insufficient illumination, saturated readings, or environmental interference, overlays based on those signals may be deemphasized, suppressed, or replaced with alternate representations derived from more reliable modalities.

[0119] The adaptive visualization system may also incorporate contextual information such as lighting conditions, surface reflectivity, material properties, or recent user interaction history. Based on these factors, the processor adjusts the type of overlay presented to optimize visibility and interpretability while maintaining spatial alignment with the underlying physical surface or object. For example, under bright ambient lighting, the system may shift from color-coded overlays to contour-based or stippled patterns that remain visible against high-luminance backgrounds. When tracking glossy or semi-reflective surfaces, overlays may preferentially rely on depth or thermal cues rather than RGB features. As the user progresses through a task, the system may transition between different visualization modes, such as initial guidance indicators, mid-task correction cues, and completion-status overlays, inferred from the user’s progress and interaction history.35Attorney Docket No. 19842-006WOU1

[0120] The system may further adjust overlay geometry and behavior based on the user’s motion or viewpoint. When the AR device detects rapid head or hand movement, overlays may temporarily simplify into thicker contours or larger glyphs to maintain legibility despite motion. Conversely, during fine detail work, overlays may become more precise, revealing higher-resolution features, micro-guidance arrows, or contour-specific shading. These adjustments affect the visual representation rather than the underlying spatial mapping or task inference. The adaptive visualization may also incorporate temporal smoothing when sensor readings fluctuate or when emphasis shifts between sensing modalities, ensuring stable and interpretable output even during sensor transitions.

[0121] In an exemplary system, the system adapts the AR visualization based on resolution changes resulting from dynamic zoom or magnification by the imaging sensor. Certain materials, patterns, or features may be indiscernible at a wide field of view but become detectable when the camera or AR device applies optical or digital zoom. As the zoom factor increases, previously invisible surface variations, such as micro-textures, pore patterns, fine cracks, residue specks, or subtle reflectance gradients, become detectable to the processor. The processor detects the change in effective spatial resolution and determines that a higher-detail visualization mode is supported. The system therefore transitions to a different overlay mode that adapts to the new context, and potentially emphasizes fine-grained guidance, such as micro-contour highlights, small-area treatment indicators, or high-resolution segmentation boundaries spatially aligned to surface detail visible only at the zoomed scale.

[0122] Conversely, when the field of view is wide and resolution per unit area is lower, the system may simplify overlays to coarse-scale regions, large coverage blocks, or global guidance cues that remain legible at distance. The processor may therefore modify the density, granularity, or spatial frequency of displayed overlays based on the capture zoom and the corresponding level of observable surface detail, ensuring that the visualization remains interpretable and appropriate for the visible level of detail. In this mode, overlays emphasize task-level guidance rather than fine surface features. This adaptive behavior allows the system to reveal high-fidelity overlays only when the imaging conditions support such precision, enabling both coarse- and fine-scale guidance within the same task.36Attorney Docket No. 19842-006WOU1

[0123] Tn another illustrative example, the system assists in cleaning a countertop where residue is detectable primarily under UV illumination. When the UV lamp integrated into the tool is active and producing reliable fluorescence readings, the AR device displays a high-opacity contamination overlay highlighting regions where residue persists. As the user wipes the surface and the residue diminishes, the fluorescence signal becomes weaker, and the system automatically transitions to depth- and reflectance-based overlays that detect moisture streaks , surface discontinuities, or incomplete drying. During rapid wiping, or periods of elevated user motion, the system simplifies the visualization to directional arrows indicating remaining target areas. As the task nears completion, the AR device increases the transparency of the overlays, revealing only subtle indicators of residual streaks or surface non-uniformities. Through this adaptive visualization approach, the system provides continuous, context-aware guidance that remains interpretable across changing sensing conditions and task stages.

[0124] In another exemplary system, the system adapts AR visualizations based on object segmentation, classification, and geometric inference. When the AR device identifies a physical object, such as a container, hand, tool, appliance, or planar surface, the system may generate overlays that are spatially anchored to that object. As the understanding of the object improves through refined segmentation, additional viewpoints, or higher-resolution sensing, the overlay may transform from a simple 2D indication to a more sophisticated 3D, contoured, or shape-aware representation.

[0125] For instance, a 2D outline of a surface may transform into a 3D surface mesh when sufficient depth, texture, or parallax information becomes available, allowing overlays to conform to curvature, edges, or internal regions of the object. Similarly, a generic rectangular label may transition into a segmented multi-region overlay once the system classifies sub-regions associated with different functions , material properties, or application zones. These transformations occur dynamically and allow the system to present overlays that increase in spatial precision as the system’s understanding of the scene improves.

[0126] This adaptive overlay evolution is particularly useful when interacting with deformable, partially occluded, or initially ambiguous surfaces, where early guidance must be simple but later guidance can be highly detailed once the system resolves object identity, geometry, or internal37Attorney Docket No. 19842-006WOU1structure. Although generically converting 2D cues to 3D overlays is known in mixed-reality systems, the present system performs such transitions selectively in response to sensor-derived confidence metrics, classification certainty, and real-time segmentation state as part of its adaptive visualization pipeline.

[0127] FIG. 12 illustrates the resolution-adaptive capabilities of the system. A user 400 scans a target surface 405 containing subtle features using an AR. imaging device 402. In a standard operational mode with a wide field of view (lx Zoom) 410, the system processes a low-resolution metric data line 412. Consequently, the corresponding AR display view 420 renders a coarse region overlay 422, providing a high-level visual summary suitable for broad navigation. However, when the system focuses on a specific area with a narrow field of view (Zoomed) 430, it captures a high- resolution metric data line 432. This enhanced data stream enables the AR display view 440 to generate a fine-grained micro-guidance overlay 442, revealing minute details and precise actionable feedback that were previously unresolved.Example 5: Active Illumination

[0128] In another embodiment, the system employs active illumination to reveal, enhance, or differentiate material characteristics that are not reliably visible under ambient lighting. The AR device or an external tool may include ultraviolet, infrared, polarized, angled, or structured-light emitters, and the processor selectively activates these illumination modes to elicit diagnostic responses from surfaces or materials. These responses may include fluorescence, reflectance changes, temperature variations, scattering patterns, or absorption signatures. Sensor data collected during such illumination is associated with the active illumination state and spatially registered to the AR world coordinate frame, and is used to generate overlays that highlight characteristics of semi- visible or invisible materials.

[0129] The system may further coordinate illumination modes with environmental context, sensing confidence, or predicted user intent. For example, if UV illumination enhances contrast for residue detection, or if infrared illumination reveals drying or absorption patterns, the system activates the corresponding mode at relevant stages of a task. Illumination selection may also depend on ambient lighting conditions, surface reflectivity, material type, or whether a coarse- or fine-scale38Attorney Docket No. 19842-006WOU1analysis is required. The processor evaluates which illumination mode yields the highest discriminative signal for the current task state, and the AR overlays adapt accordingly, changing style, opacity, spatial emphasis, or structure based on which modality most effectively reveals the underlying surface condition.

[0130] In certain embodiments, illumination control is further coordinated with zoom level or field-of-view changes from the AR device or external camera. At a wide-field view, the processor may activate lighting modes optimized for large-scale or coarse material classification (e.g., visible or polarized illumination), producing broad guidance overlays. When the imaging sensor transitions to a zoom-in state, adding detail and resolving micro-features, the processor may activate a different illumination mode, such as UV or IR, that enhances fine-scale surface characteristics only visible at increased magnification. Because illumination and zoom jointly affect the spatial scale at which material features are observable, the AR overlays dynamically transition from coarse indicators in wide view to high-resolution guidance during zoom, reflecting the combined sensing richness provided by coordinated illumination and magnification.

[0131] In some embodiments, the processor synchronizes illumination switching and sensor capture to acquire alternating frames under different lighting modes. Each frame is time-stamped and spatially aligned within the AR world frame, allowing direct comparison of measurements obtained under different illumination conditions. By comparing these temporally aligned frames, the system distinguishes material classes, surface moisture levels, chemical residues, or absorption gradients that cannot be reliably identified under a single illumination state. The AR overlays reflect these distinctions, guiding the user toward untreated regions, improperly cleaned areas, or surfaces requiring additional application of product or time to fully react.

[0132] FIGS. 13A-13B demonstrate the system's coordinated illumination capabilities. In FIG. 13A, a user 500 views a target surface 505 under ambient visible light 508. The AR glasses 502 operate with a wide field of view capture 515, and the coordinated illumination controller & processor 550 determines that no specialized lighting is currently needed. Consequently, the AR display view 520 presents a coarse guidance overlay 522. In contrast, FIG. 13B depicts the system transitioning to a high-precision mode. The controller 550 dims the ambient light and activates the integrated UV emitter 510, projecting UV illumination rays 532 onto the surface. Simultaneously,39Attorney Docket No. 19842-006WOU1the sensors switch to a zoomed field of view capture 535. This configuration reveals fluorescent micro-residue 542 invisible to the naked eye. The corresponding AR display view 540 then generates a fine-grained detection overlay 544, precisely mapping the residue for targeted cleaning or treatment.Example 6: Dynamic Guidance

[0133] In another embodiment, the system dynamically adapts its guidance strategy based on an assessment of the user’s skill level, historical behavior, current performance, and the evolving state of the task. The processor receives sensor data including motion patterns, tool trajectories, materialuse statistics, surface-state changes, timing intervals, error detections, or environmental conditions, and analyzes these signals over time and across task-relevant spatial regions to infer whether the user is performing the task confidently, hesitantly, inaccurately, or with inconsistent efficiency. Using this inference, the AR device modulates the level of guidance, the type of indicators displayed, and the granularity of feedback presented to the user.

[0134] The system may identify novice-like behavior, such as irregular stroke patterns, excessive pauses, inconsistent pressure or contact, or inefficient product consumption, and respond by increasing the explicitness of guidance cues. These cues may include step-by-step directional arrows, suggested motion trajectories, pacing indicators, ideal coverage patterns, or alerts for underapplication, over-application, or incorrect tool orientation. Conversely, when the user demonstrates expert-like patterns, smooth trajectories, consistent coverage rate, stable orientation, or optimized material use, the system may reduce the intrusiveness of overlays, transitioning from directive guidance to confirmatory or summary indicators, allowing faster, uninterrupted workflow.

[0135] In some embodiments, the processor identifies specific error signatures associated with the task. For example, when assisting with application of a topical product, the system may recognize characteristic patterns of streaking, incomplete blending, repeated area targeting without sufficient material, or incorrect hand-tool distance. These error signatures are detected by correlating motion behavior, material-state signals, and spatial context. The AR overlay may then selectively highlight areas requiring correction, provide real-time course adjustments, or display predictive 40Attorney Docket No. 19842-006WOU1indicators showing where the user’s current trajectory will likely result in error before the error occurs.

[0136] The system may also generate personalized guidance profiles over time. By analyzing historical sessions including time-to-complete, coverage uniformity, product usage efficiency, number of corrections required, and user-specific motion kinematics, the processor adjusts future guidance levels for the same or similar tasks. For example, a user who consistently applies excessive product may receive consumption-efficiency feedback earlier in the task, whereas a user who works too quickly may receive pacing corrections or stability cues. The AR system thereby becomes increasingly tailored to each user’s unique behavior while preserving consistent task objectives and safety constraints

[0137] Finally, in certain embodiments, the system adapts guidance based not only on user behavior but also on real-time task conditions. If sensors detect deteriorating lighting, partial occlusion, rapid user motion, or ambiguous material-state signals, the system may temporarily increase the specificity or redundancy of overlays to ensure safe and accurate task execution. This may include combining multiple overlay cues, increasing visual persistence, or reintroducing higher- level guidance temporarily. When conditions stabilize, guidance may revert to a less intrusive mode. This dynamic, context-aware modulation allows the system to deliver the appropriate level of assistance at each moment without overwhelming or distracting the user.

[0138] FIGS. 14A-14B illustrate the system's adaptive guidance logic. A skill assessment & guidance controller 650 monitors the kinematic behavior of a user's hand 600 as they manipulate a handheld tool 602 over a target surface 605. In FIG. 14A (High Assistance Mode), the controller detects an erratic motion path 610 characteristic of novice or unstable input. In response, the AR display view 620 presents explicit directional guide arrows 622 to stabilize the stroke trajectory and a correction warning icon 624 to flag significant deviations. Conversely, in FIG. 14B (Low Assistance Mode), the user demonstrates a smooth motion path 630 indicative of expert or stable control. Recognizing this proficiency, the AR display view 640 transitions to a minimalist interface, displaying only a subtle progress bar / perimeter outline 642 and a stability confirmed icon 644, thereby validating performance without interrupting the user's established workflow.41Attorney Docket No. 19842-006WOU1Example 7: Shared-Autonomy Configuration

[0139] In another embodiment, the system operates in a shared-autonomy configuration in which a human user performs natural, gross-motor movements of a handheld tool, instrument, applicator, or device, while the processor executes fine-scale corrective actuation based on real-time AR perception, sensor fusion, and predictive modeling. In this configuration, the AR device continuously estimates the spatial relationship between the tool, the target surface, and the user’s motion, and computes an idealized local micro-adjustment profile, such as a micro-trajectory, force profile, or actuation pattern that improves precision, stability, and safety without removing the human from the control loop. The tool or platform incorporates one or more actuators, such as micro-positioning motors, torque or speed regulators, variable-flow dosing valves, oscillation control elements, gimbal stages, or adaptive illumination assemblies that may be driven automatically by the processor to refine, stabilize, or constrain the user’s motion.

[0140] The AR device captures multi-modal sensor data, including RGB and depth imagery, inertial motion data, material-state measurements, predictive coverage maps, UV or 1R feedback, or force and torque signals from integrated or external sensors. These measurements are fused into a unified coordinate frame and used to infer the intended task path, the ideal tool orientation, predicted task outcomes, and real-time deviations between the user’s actual movement and an optimal movement profile. Deviation metrics may include angular error, depth error, force mismatch, timing offset, or predicted outcome variance. When deviations exceed a threshold, or when the AR system predicts that the current trajectory will result in error, the processor issues commands to adjust one or more actuators. Examples include micro-correction of tool angle, automated modulation of torque or speed to prevent material damage, fine-scale stabilization to counteract human tremor, depth-limit enforcement, or dynamic dosing adjustments during material application.

[0141] The user maintains voluntary control of the macro-motion such as placing the tool at a location, sweeping across a surface, or guiding a surgical or mechanical instrument while the system supplies precisely timed micro-actuation. This shared-control architecture allows human dexterity, judgment, and environmental awareness to combine with machine-level precision and repeatability. The AR overlays displayed to the user reflect both the user-driven motion and the system-generated42Attorney Docket No. 19842-006WOU1micro-corrections, making the assistance transparent and interpretable, and providing real-time visualization of intended trajectories, corrected paths, force indicators, predicted outcomes, and safety boundaries.

[0142] In some embodiments, the tool includes a micro-gimbal or piezoelectric stage that provides lateral or angular correction of the tool tip to maintain perpendicularity, follow an ideal trajectory, or compensate for user tremor. In other embodiments, a torque regulator modulates rotational output dynamically based on substrate hardness estimation, screw drive conditions, or predicted breakthrough depth. For dispensing or spraying tools, the system may operate dosing valves, atomization controls, or nozzle angle actuators to ensure accurate material deposition even when user motion is inconsistent. For surgical or medical instruments, the system may enforce safe boundaries, suppress involuntary tremor, maintain optimal cutting depth, or provide microstabilization around delicate structures, without preventing intentional user motion outside those boundaries.

[0143] Because the AR system continuously models both the task state and the material state, micro-actuation can also be driven by the sensed effect of the tool on the environment. For example, if image data indicate insufficient surface removal, incomplete material deposition, or an undesirable change in texture or reflectance, the processor may temporarily increase or decrease actuator output, alter the micro-trajectory, or modulate contact force. Likewise, predictive modeling can determine that a user’s current motion will cause misalignment or damage several milliseconds into the future; the system may then proactively adjust actuators to avoid such outcomes before irreversible interaction occurs.

[0144] In certain embodiments, micro-actuation is coordinated with adaptive zoom, focal adjustment, or illumination control from a pan-tilt-zoom (PTZ) camera or sensor array, ensuring that perception and actuation pathways remain synchronized. Perceptual resolution, sensing modality, and actuation sensitivity may be adjusted together based on task phase or spatial scale. The shared- autonomy system thus forms a closed loop in which perception drives actuation, actuation modifies the environment, and updated sensor measurements refine the next actuation command, all while the user remains in the primary motion-control role. This architecture substantially enhances precision,43Attorney Docket No. 19842-006WOU1safety, and outcome quality across tasks ranging from material application and drilling to surgical preparation and robotic-assisted manipulation.

[0145] FIGS. 15-17 illustrate the shared-autonomy architecture. In this configuration, the human user retains control over the general movement and decision-making, while the system automatically executes fine-scale adjustments to ensure precision and stability. As shown in FIG. 15, a user 702 performs a task on a target surface / object 720 using an active smart tool 710. The system aggregates multi-modal data from the user's hand 1213, tool embedded sensors 1212a -1212c, the AR head-mounted display (HMD) 704 (via HMD data bus 705), and an external sensor node 730 (via external sensor data path 732). This collective data is processed by the multi-sensor fusion & shared autonomy processor 700, which generates two synchronized outputs. First, as depicted in FIG. 16 via the adaptive overlay visualization data path 707, the user sees an augmented view where their actual gross motion path 740 is contrasted with a system-optimized ideal path 742, supplemented by a material state feedback icon 744 and a micro-correction active indicator 746. Second, as detailed in the cross-section of FIG. 17, the processor transmits a real-time microactuation command signal 714 to the smart tool handle 750. This signal drives integrated microactuators 755 to adjust the movable smart tool tip 752 along precise micro-adjustment vector arrows 758, thereby refining the manual input to ensure precision and safety.Magnetic Fields

[0146] In another embodiment, the system is configured to detect, reconstruct, and visualize time-varying magnetic fields present in a physical environment and to present spatially registered augmented reality guidance derived from temporal and spatial changes in such magnetic fields. One or more magnetic field sensors are provided, which may include Hall-effect sensors, magnetoresistive sensors, fluxgate magnetometers, vector magnetometers, gradiometers, optically pumped magnetometers, or other magnetic sensing technologies capable of measuring magnetic field magnitude, direction, gradient, or flux density as a function of time. The magnetic field sensors may be integrated into a handheld tool, embedded within a machine or device, mounted within the environment, or worn by the user. Each sensor provides magnetic field measurements together with44Attorney Docket No. 19842-006WOU1positional or orientation information sufficient to spatially associate the measurements with the physical environment.

[0147] Measurements from the magnetic field sensors are received by the processor and transformed into the spatial coordinate frame established by the augmented reality device. The processor further analyzes temporal sequences of magnetic field measurements to estimate a dynamic spatial distribution of the magnetic field, including changes in field geometry, magnitude, direction, or symmetry over time. Such analysis may incorporate temporal filtering, phase analysis, pattern recognition, or model-based reconstruction to infer spatially continuous or region-based magnetic field behavior from discrete measurements, and to identify evolving magnetic field behaviors associated with system operation, load conditions, control states, or environmental interactions. In some embodiments, the processor employs one or more inference models, including machine-learning or statistical models, to interpret temporal patterns in the magnetic field measurements, identify expected or anomalous field behaviors, and infer system states or predicted outcomes associated with the observed magnetic field evolution.

[0148] The augmented reality device renders one or more AR overlays representing the dynamic magnetic field distribution in spatial alignment with physical objects, surfaces, or volumes in the real environment. The overlays may visualize magnetic field evolution using animated vectors, time-dependent streamlines, pulsating or shifting contour regions, color-coded gradients indicating flux intensity changes, or volumetric representations that convey both magnitude and temporal variation. The overlays are anchored within the AR world coordinate frame such that magnetic field features appear fixed relative to the corresponding physical structures as the user moves. The visualization may adapt in resolution, density, or rendering style based on zoom level, viewing distance, operating state of the observed system, or confidence in the underlying measurements.

[0149] In addition to visualization, the system generates contextual guidance based on detected changes in magnetic field behavior. For example, the processor may identify flux asymmetries, unexpected leakage, temporal instabilities, phase imbalances, or deviations from an expected magnetic field signature associated with normal operation. Based on such analysis, the AR system may present guidance cues indicating potential inefficiencies, fault conditions, misalignment,45Attorney Docket No. 19842-006WOU1improper configuration, or regions requiring inspection or adjustment. These guidance cues are spatially registered to the physical locations associated with the detected magnetic anomalies, assisting users in tasks such as diagnosing electromechanical systems, tuning adaptive magnetic components, verifying correct assembly or operation, or understanding the interaction between magnetic fields and physical structures during operation.

[0150] In some embodiments, the dynamic magnetic field visualization and guidance are integrated with other sensor-derived information, including visual, depth, thermal, inertial, electrical, or material-state data, such that magnetic field evolution is correlated with changes in mechanical motion, temperature, load, or applied control signals. The processor temporally aligns these heterogeneous data streams within the AR world frame, enabling the AR system to present a unified, time-synchronized representation of otherwise invisible physical phenomena, and allowing users to observe how magnetic field behavior responds to operational conditions in real time.

[0151] In certain embodiments, the processor may further couple dynamic magnetic field analysis with actuator control. For example, when a tool, machine, or system includes one or more controllable actuators, the processor may automatically adjust orientation, spacing, operating parameters, or control inputs in response to detected magnetic field changes, while the AR device presents corresponding guidance, confirmation, or warning overlays. Such actuation may provide fine-scale corrective adjustments while the user retains control of gross-motor actions or high-level decisions, improving efficiency, maintaining stability, preventing undesirable magnetic interactions, or guiding human intervention.

[0152] Through these mechanisms, the system enables users to perceive, interpret, and interact with dynamically changing magnetic fields as spatially contextualized elements of the physical environment, extending augmented reality beyond static visualization to real-time understanding and guidance based on evolving electromagnetic behavior.

[0153] FIG. 18 illustrates an augmented reality environment for visualizing invisible physical fields. A user

[1804] wearing the AR head-mounted display interacts with an electric motor / stator

[1802] using a handheld sensor probe

[1806] , Real-time measurement data is wirelessly transmitted from the sensor directly to a local Edge Al processor

[1820] , which aggregates sensor inputs and performs low-latency inference. The processed visualization data is then transmitted to a wireless 46Attorney Docket No. 19842-006WOU1module

[1812] of the headset, which renders dynamic magnetic flux lines

[1808] as a synchronized virtual overlay. Upon detecting an anomaly, the system generates a floating AR alert box

[1814] (e.g., "FLUX LEAKAGE") and may further project a guidance arrow

[1816] directly onto the affected region to precisely pinpoint the location of the leakage or the specific component requiring adjustment.

[0154] As used herein, “AR” or “Augmented Reality” means a system for providing enhanced digital overlays to a user’s environment. While the examples described herein primarily refer to AR glasses, the term AR is not limited to glasses and may also include other wearables, such as smartwatches, tablets, and other devices having holographic displays. The holographic displays may be Mixed Reality (MR) headsets, table-top hologram projections, or other floating variations that provide similar functionality and visualization capabilities.

[0155] As used herein, “near real-time” or “real time” describes a process that occurs or a system that operates to produce a given result with a slight but acceptable delay between the occurrence of an event, such as an acquisition of or update to relevant data, and when the given result is produced. In the context of the present disclosure, a slight but acceptable delay is in the range of about 250 milliseconds.

[0156] Various modifications and additions can be made without departing from the spirit and scope of this disclosure. Features of each of the various embodiments described above may be combined with features of other described embodiments as appropriate in order to provide a multiplicity of feature combinations in associated new embodiments. Furthermore, while the foregoing describes a number of separate embodiments, what has been described herein is merely illustrative of the application of the principles of the present disclosure. Additionally, although particular methods herein may be illustrated and / or described as being performed in a specific order, the ordering is highly variable within ordinary skill to achieve aspects of the present disclosure. Accordingly, this description is meant to be taken only by way of example, and not to otherwise limit the scope of this disclosure.

[0157] Exemplary embodiments have been disclosed above and illustrated in the accompanying drawings. It will be understood by those skilled in the art that various changes, omissions and47Attorney Docket No. 19842-006WOU1additions may be made to that which is specifically disclosed herein without departing from the spirit and scope of the present disclosure.48Attorney Docket No. 19842-006WOU1

Claims

What is claimed is:

1. A system for providing guidance to a user performing a task, comprising: at least one sensor configured to detect a physical quantity associated with the task; a wireless communication module configured to transmit data from the sensor, wherein the data includes a set of values of the physical quantity and a set of spatial parameters corresponding to locations associated with the set of values of the physical quantity detected by the sensor; and an augmented reality (AR) device having a camera, a receiver, a processor, and a display, wherein the sensor is external to the AR device and wherein the processor includes instructions to: process data received from the sensor, generate a representation of a virtual space, wherein the virtual representation of the space includes the locations associated with the set of values of the physical quantity detected by the sensor, generate an overlay reflecting a status for one or more areas of the virtual space based on the set of values of the physical quantity and the set of spatial parameters, and provide real-time guidance to the user for the task by displaying the overlay in conjunction with the virtual space on the display.

2. The system of claim 1, wherein the processor includes instructions to determine a spatial transformation between a coordinate frame of the sensor and a coordinate frame of the AR device, and wherein generating the overlay includes converting the set of spatial parameters into the coordinate frame of the AR device using the spatial transformation.

3. The system of claim 2, wherein determining the spatial transformation comprises detecting a fiducial marker, shape, or pose of the sensor using the camera of the AR device.

4. The system of claim 1, wherein the sensor is external to the AR device.

5. The system of claim 1, wherein the sensor is integrated into a product container, a wearable device, a tool, or industrial equipment.

6. The system of claim 1, wherein the wireless communication module utilizes Bluetooth or Wi-Fi.

7. The system of claim 1, wherein the overlay includes heatmaps with different indicators for treated and untreated areas of an object on which the task is performed.49Attorney Docket No. 19842-006WOU18. The system of claim 1, wherein the sensor detects an amount of product dispensed and the processor is configured to provide alerts for insufficient or excessive product usage.

9. The system of claim 1, wherein the sensor is configured to measure one or more of a flow rate of a liquid, a surface coverage of a product, temperature, humidity, motion, or orientation of a container or device.

10. The system of claim 1, further comprising a mobile or cloud-based platform for data storage, analysis, and enhanced visualization.

11. The system of claim 1, wherein the AR device includes modules configured to determine product shape, orientation, and labels based on image data received from the camera.

12. The system of claim 7, wherein the object is a floor and the physical quantity is UV reflectance.

13. The system of claim 7, wherein the object is hair of the user and the physical quantity is a degree of light polarization.

14. The system of claim 1, wherein the overlay is displayed in relation to an object related to the task.

15. The system of claim 4, wherein the processor is configured to establish a unified spatial coordinate frame using sensor data from the AR device, including at least depth information, inertial measurements, or scene-reconstruction techniques, and to register measurements from the sensor to the unified spatial coordinate frame.

16. The system of claim 15, wherein the external sensor is associated with a calibration transform defining a spatial relationship between the external sensor and a corresponding tool, package, or object, and wherein the processor applies the calibration transform to map measurements from the sensor to coordinates in the unified spatial coordinate frame.

17. The system of claim 15, wherein the processor is further configured to receive measurements from one or more environmental sensors and transform those measurements into the unified spatial coordinate frame based on a known installation geometry or a dynamically determined pose of the environmental sensor.

18. The system of claim 1, wherein the processor performs multi-modal sensor fusion by combining asynchronous measurements from at least two modalities selected from optical, inertial, depth, thermal, ultraviolet, chemical, force, or flow sensors to generate a temporally coherent representation of a surface or environment.50Attorney Docket No. 19842-006WOU119. The system of claim 1, wherein the processor executes a machine-learning model configured to interpret fused sensor data to estimate a property of a material applied to a surface, including material coverage, distribution uniformity, or absorption state.

20. The system of claim 19, wherein the processor generates, for display by the AR device, a spatially registered overlay representing information derived from the fused sensor data, including at least one of: a coverage map, a material-density gradient, a predicted flow or dispersion pattern, or an indicator of surface regions requiring additional treatment.

21. The system of claim 1, wherein the sensor includes a dispensing-rate sensor associated with a sunscreen container, and wherein the processor determines sunscreen coverage on a user’s skin by combining dispensing-rate data, hand-motion data, and image data captured by the AR device.

22. The system of claim 19, wherein the processor determines an estimated amount of material required to complete a treatment based on the fused sensor data, and generates guidance to reduce excess material consumption.

23. The system of claim 1, wherein the processor is configured to analyze motion data of a user’s hand, head, or tool to predict an intended action or target region prior to completion of the action.

24. The system of claim 1, wherein the processor determines a predicted intent by correlating temporal sequences of sensor data including at least one of: tool trajectory, dispensing rate, surface reflectance, thermal output, or airborne-compound detection.

25. The system of claim 24, wherein the processor generates predictive overlays based on the predicted intent, the overlays including at least one of: a projected tool path, a recommended region of material application, or a caution indicator corresponding to a predicted undesirable outcome.

26. The system of claim 25, wherein the processor updates the predictive overlays in response to changes in detected motion of the user or environmental conditions.

27. The system of claim 24, wherein the predicted intent is further based on environmental context derived from depth sensing, surface geometry, illumination conditions, or measurements from thermal, ultraviolet, or chemical sensors.

28. The system of claim 25, wherein the processor provides corrective guidance prior to material application when the predictive overlays indicate insufficient coverage, improper alignment, or deviation from a desired application pattern.51Attorney Docket No. 19842-006WOU129. The system of claim 1 , wherein the processor is configured to receive a time series of sensor measurements, to determine a temporal evolution of a material applied to a surface, and to generate temporally aligned sensor data.

30. The system of claim 1, wherein the processor identifies a presence or concentration of a material that is invisible or semi-visible in any single measurement frame by evaluating changes in reflectance, fluorescence, thermal output, or surface texture across multiple frames.

31. The system of claim 29, wherein the processor determines one or more dynamic material properties including absorption rate, diffusion rate, drying rate, dwell time, or chemical reaction progression based on the temporally aligned sensor data.

32. The system of claim 29, wherein the processor predicts a future state of the material by extrapolating a temporal progression of sensor measurements.

33. The system of claim 1, wherein the AR device generates and displays a temporal overlay indicating at least one of: an area of insufficient absorption, a region at risk of uneven drying, or a surface predicted to require additional treatment.

34. The system of claim 33, wherein, when the AR device displays the temporal overlay, measurements are fused from at least two modalities selected from RGB imaging, UV reflectance, infrared imaging, thermal sensing, airborne-compound detection, or dispensing-rate sensing.

35. The system of claim 1, wherein the processor determines absorption and distribution of a transparent skincare serum by analyzing temporal changes in reflectance under visible or ultraviolet illumination together with dispensing-rate data.

36. The system of claim 1, wherein the sensor includes a dispensing-rate sensor associated with a hair-dye container, and wherein the processor predicts a next region of hair to be treated based on applicator motion, dispensing data, and segmentation of the user’s scalp.

37. The system of claim 1, wherein the processor adjusts an AR overlay based on a confidence level associated with one or more sensing modalities, including RGB imaging, depth sensing, ultraviolet sensing, thermal sensing, chemical sensing, or dispensing-rate sensing.

38. The system of claim 1, wherein the processor increases visual prominence of overlays derived from sensor data determined to be reliable and decreases visual prominence of overlays derived from sensor data affected by occlusion, motion blur, insufficient illumination, or interference.52Attorney Docket No. 19842-006WOU139. The system of claim 1, wherein the processor modifies a style, opacity, contrast, or geometric representation of an overlay based on ambient lighting conditions, surface reflectivity, material properties, or user interaction history.

40. The system of claim 1, wherein the processor transitions between different visualization modes, including initial-guidance overlays, mid-task correction overlays, and completion-status overlays, based on an inferred stage of a user task.

41. The system of claim 1, wherein the processor adapts a size, thickness, or complexity of the overlay in response to detected head motion or hand motion to maintain visual legibility.

42. The system of claim 1, wherein the processor applies temporal smoothing to the overlays when the system transitions between sensing modalities or when sensor readings fluctuate.

43. The system of claim 1, wherein the processor generates an adaptive contamination overlay based on UV fluorescence during detection of surface residue, and transitions to moisture- or reflectance-based overlays as the residue diminishes.

44. The system of claim 1, wherein the processor modifies a type, form, or informational content of an AR overlay based on a zoom state of the camera of the AR device, such that the overlay presented at a wide-field view differs functionally from the overlay presented at a zoomed-in view.

45. The system of claim 1, wherein the processor selects between a coarse-scale guidance mode and a fine-detail guidance mode based on a zoom level of the camera, the fine-detail guidance mode providing task indicators that are not displayed in the coarse-scale mode.

46. The system of claim 1, wherein the processor controls one or more illumination sources including ultraviolet, infrared, polarized, or structured-light emitters to reveal material characteristics not visible under ambient lighting.

47. The system of claim 1, wherein the processor identifies a material property based on a response to an illumination mode, including fluorescence, reflectance change, scattering pattern, absorption profile, or thermal emission.

48. The system of claim 1, wherein the processor selects an illumination mode based on environmental conditions, material type, or a predicted user intent.

49. The system of claim 1, wherein the processor is configured to receive sequential frames under differing illumination modes and compare the sequential frames to distinguish between material classes or surface conditions.53Attorney Docket No. 19842-006WOU150. The system of claim 1, wherein the overlay changes in type, appearance, or informational content based on which illumination mode is active.

51. The system of claim 1, wherein the processor is configured to activate one or more illumination modes of the AR device based on a zoom state of the AR device or an external camera, such that a first illumination mode is activated for capturing a wide-field of view and a second illumination mode is activated for capturing a zoom-field of view.

52. The system of claim 1, wherein the processor modifies the overlay based on a combined condition of illumination mode and zoom level to present material indicators that differ between low-magnification and high-magnification viewing.

53. The system of claim 1, wherein the processor activates UV illumination to detect fluorescent residue and transitions to visible, polarized, or infrared illumination to analyze moisture or streaking as the residue diminishes.

54. The system of claim 1, wherein the processor adjusts a level or type of AR guidance based on an inferred user skill level determined from motion patterns, tool trajectories, material -use statistics, or historical performance data.

55. The system of claim 1, wherein the processor increases granularity or intrusiveness of guidance cues when detecting irregular stroke patterns, inconsistent coverage, excessive product use, or repeated corrective motions.

56. The system of claim 1, wherein the processor decreases the prominence of guidance cues when the processor determines that the user demonstrates stable tool orientation, uniform coverage rate, efficient material use, or smooth trajectory patterns.

57. The system of claim 1, wherein the processor predicts a likely error based on real-time motion or orientation analysis and generates an AR overlay indicating a corrective trajectory before the error occurs.

58. The system of claim 1, wherein the processor maintains a user-specific performance history and adjusts future guidance parameters based on patterns learned from previous tasks.

59. The system of claim 1, wherein the processor increases the redundancy of AR overlays in response to a determination of degraded sensing conditions including low lighting, occlusion, motion blur, or sensor ambiguity.

60. The system of claim 1, wherein the processor transitions between step-by-step guidance and summary-level feedback based on a combined assessment of user behavior and current task state.54Attorney Docket No. 19842-006WOU161 . The system of claim 1, wherein the processor operates one or more micro-actuators integrated into a handheld tool associated with the task to apply fine-scale lateral, angular, or axial corrections to a tool tip or instrument path based on deviations between an estimated current trajectory and a projected trajectory determined from fused sensor data.

62. The system of claim 61, wherein the processor identifies high-frequency involuntary user motion and commands a stabilization mechanism within the handheld tool to suppress tremor, vibration, or jitter.

63. The system of claim 1, wherein the processor adjusts an actuator output that includes torque, rotation speed, dosing flow rate, nozzle angle, contact force, depth limit, or oscillation amplitude in response to a detected or predicted material state derived from multi-modal sensor inputs.

64. The system of claim 1, wherein the processor predicts a future error condition associated with a user-driven motion and preemptively actuates a mechanical, dosing, or positioning subsystem to avoid the error before it occurs.

65. The system of claim 1, wherein the processor enforces virtual spatial boundaries by restricting actuator output or counteracting user movement when the user moves a tool or instrument outside a safe or task-appropriate region displayed in the AR overlay.

66. The system of claim 1, wherein the processor synchronizes actuation commands with AR perception parameters by adjusting micro-actuation output based on changes in zoom, focal distance, illumination intensity, or sensor confidence determined by the processor.

67. The system of claim 1, wherein the processor selects or updates an actuation profile for a tool or instrument based on historical user performance, detected material characteristics, or real-time task state, such that actuator behavior becomes increasingly optimized for the user over time.

68. The system of claim 1, wherein the processor concurrently controls a plurality of actuators and external pan -tilt-zoom (PTZ) actuators, to jointly refine a human-driven action and achieve a target precision, depth, trajectory, or material distribution pattern.

69. The system of claim 1, further including one or more magnetic sensors, wherein the processor is configured to receive magnetic field measurements from the one or more magnetic sensors and to reconstruct a spatial representation of magnetic field magnitude, direction, or flux density within a region of interest aligned to the augmented reality coordinate frame.55Attorney Docket No. 19842-006WOU170. The system of claim 69, wherein the augmented reality device displays a magnetic field overlay comprising directional vectors, streamlines, contour maps, or color-coded gradients representing magnetic field characteristics relative to physical objects or surfaces.

71. The system of claim 69, wherein the processor analyzes the reconstructed magnetic field to identify regions of interest, anomalies, or transitions in field strength or direction, and generates augmented reality guidance indicating preferred tool placement, orientation, or regions to avoid.

72. The system of claim 69, wherein magnetic field measurements are fused with visual, depth, thermal, inertial, electrical, or material-state data to generate a combined augmented reality representation of visible and invisible physical phenomena.

73. The system of claim 70, wherein the processor dynamically adjusts a density, scale, resolution, or visual style of the magnetic field overlay based on viewing distance, zoom level, sensor confidence, or user interaction.

74. The system of claim 69, wherein the processor employs one or more inference models, including machine-learning or statistical models, to interpret temporal patterns in magnetic field measurements, identify expected or anomalous magnetic field behaviors, and infer system states or predicted outcomes, and wherein the augmented reality device presents guidance or feedback based on the inferred states.

75. The system of claim 74, wherein the processor controls one or more actuators associated with a tool or device in response to inferred magnetic field behavior, while presenting augmented reality feedback corresponding to the actuator adjustment.

76. The system of claim 25, wherein the task is a welding operation, and wherein the processor is configured to predict a timing and location of a welding arc ignition based on the tool trajectory and to darken a portion of the display prior to the predicted ignition to protect the user's eyes.

77. The system of claim 1, wherein the sensor is an aroma analyzer configured to detect volatile organic compounds (VOCs), and wherein the processor is configured to generate the overlay as a volumetric representation of a diffusion profile or intensity field of the detected VOCs relative to a source obj ect.

78. The system of claim 1, wherein the sensor is integrated into an autonomous robotic device performing the task, and wherein the overlay indicates regions treated by the robotic device and regions requiring manual intervention by the user.56Attorney Docket No. 19842-006WOU179. The system of claim 1, wherein the processor records the spatial parameters and user corrections during the task to update a navigation or path-planning model for an autonomous robotic device.

80. The system of claim 1, further comprising a wearable haptic interface, wherein the processor is configured to trigger a haptic cue when the detected physical quantity or a tool trajectory deviates from a target parameter.

81. A method for assisting a user in performing a physical task in a target region, comprising: receiving, by a processor, sensor data from an augmented reality device and from one or more sensors associated with a tool or instrument, the sensor data comprising at least image data, depth measurements, inertial data, or material-state information; generating, by the processor, a spatial representation of the tool, the target region, and detected motion of the user by fusing the sensor data into a unified coordinate frame; determining, by the processor, a preferred tool parameter related to the task based on the spatial representation and predictive modeling of an intended task outcome, wherein the preferred tool parameter includes trajectory, depth, orientation, force profile, torque profile, or dosing profile; detecting, by the processor, a deviation between detected user-driven motion of the tool and the preferred tool parameter; actuating, via signal from the processor, at least one actuator associated with the tool to apply a corrective adjustment that refines the user-driven motion, stabilizes the tool, alters an output characteristic, or prevents an anticipated error; and displaying, by the augmented reality device, one or more AR overlays representing the preferred tool parameter, the detected deviation, or the applied corrective adjustment.

82. The method of claim 81, wherein actuating the at least one actuator comprises applying microscale lateral, angular, or axial adjustments to a tool tip to maintain alignment with a preferred trajectory.

83. The method of claim 81, wherein detecting the deviation includes identifying high-frequency involuntary motion of the user, and the actuating includes suppressing tremor or vibration.

84. The method of claim 81, wherein determining the preferred parameter further comprises estimating a material state of a surface or substrate, and wherein the actuating includes adjusting torque, speed, dosing, contact force, or oscillation amplitude based on the estimated material state.57Attorney Docket No. 19842-006WOU185. The method of claim 81, wherein detecting the deviation includes predicting an error condition by extrapolating a user-driven trajectory, and the actuating includes applying a corrective adjustment prior to occurrence of the error condition.

86. The method of claim 81, further comprising defining, by the processor, a virtual boundary associated with a safe or task-appropriate region, and wherein the actuating includes restricting or counteracting user-driven motion that would cause the tool to cross the virtual boundary.

87. The method of claim 81, wherein actuating the actuator is synchronized with adjustments to AR perception parameters, including zoom level, illumination intensity, focal distance, or sensor confidence, such that micro-actuation output is adapted to current sensing conditions.

88. The method of claim 81, wherein actuating the at least one actuator comprises issuing coordinated control signals to multiple actuators including micro-positioners, torque or speed regulators, dosing mechanisms, or external pan-tilt-zoom (PTZ) or illumination actuators, to jointly refine the user-driven motion.

89. The method of claim 81, further comprising updating, by the processor, an actuation profile based on historical user performance or prior task outcomes, such that corrective adjustments are personalized over time.

90. A system for providing guidance to a user performing a task, comprising: a depth camera positioned to capture spatial data of an area; a sensor configured and positioned to detect a physical attribute in the area; an augmented reality (AR) device in communication with the depth camera and sensor, the AR device including a camera positioned to capture images of the area; a display system configured to provide real-time visual overlays in a field of view of the AR device, wherein the visual overlays include guidance for tools visible within the area based on data for the physical attribute from the sensor, the spatial data from the depth camera, and the images of the area.

91. A method for providing guidance to a user performing a task, comprising: receiving data from a sensor configured to detect a physical quantity associated with the task; transmitting data from the sensor to an augmented reality (AR) device, wherein the data includes a set of values of the physical quantity and a set of spatial parameters corresponding to locations associated with the set of values of the physical quantity detected by the sensor, and wherein the AR device includes a camera configured to capture images of the locations58Attorney Docket No. 19842-006WOU1associated with the set of values of the physical quantity detected by the sensor, a receiver, a processor, and a display; processing data received from the sensor and the images captured by the camera; generating a representation of a space, wherein the representation of the space includes the locations associated with the set of values of the physical quantity detected by the sensor; generating an overlay reflecting a status for one or more areas of the representation of the space based on the set of values of the physical quantity and the set of spatial parameters; and providing real-time guidance to the user for the task by displaying the overlay in conjunction with the representation of the space on the display.59Attorney Docket No. 19842-006WOU1

Images