Augmented reality content stabilization

Abstract

A system and method for presenting augmented reality (AR) content stabilized on a see-through display, said method comprising: receiving, at a computer in a vehicle, AR content, said AR content being generated based on an initial position of said see-through display; following reception of said AR content and before displaying AR content, receiving an updated position of said see-through display; comparing said initial position to said updated position to determine a change in position; and displaying said AR content on said see-through display, said displayed AR content being adjusted based at least in part on said change in position.

Description

AUGMENTED REALITY CONTENT STABILIZATIONREFERENCE TO RELATED APPLICATION

[0001] This application is based on US Provisional Application No. 63 / 754,752, filed February 6, 2025, and is incorporated herein by reference in its entirety.FIELD OF INVENTION

[0002] The present invention relates, generally, to augmented reality (AR) display systems and content rendering, and, more specifically, to techniques for stabilizing and presenting AR content on see-through displays in dynamic environments such as aircraft cockpits and other vehicles.BACKGROUND

[0003] Augmented reality systems are increasingly used to improve situational awareness for operators of vehicles, including pilots of high-performance aircraft. In such systems, virtual content (e.g., symbology, navigation cues, training overlays, targeting information, and / or hazard indications) is presented within the operator’s field of view (FOV) via a see-through display so that the operator can view both the physical environment and the virtual overlay concurrently.

[0004] In dynamic environments, however, stabilizing virtual content within the operator’s perceived FOV can be difficult. For example, in an aircraft cockpit, vibration, turbulence, g-forces, rapid maneuvers, and / or intentional head movement can cause rapid changes in the position and orientation of a head-worn display. If AR content is rendered or positioned based on a display pose that is even slightly outdated, the AR content may appear to “swim,” jitter, or drift relative to the physical environment, which can distract the operator and degrade usability and safety.

[0005] The stabilization challenge can be compounded by end-to-end latency in an AR content pipeline. Latency may be introduced by sensor acquisition, processing, rendering, encoding / decoding, wireless transmission, buffering, and display refresh timing. In some implementations, AR content is at least partly generated by a remote computer (e.g., a groundsystem) and delivered to the vehicle, which may further increase latency. Because a vehicle and head-worn display can move meaningfully during the latency interval, content generated for an initial pose may be incorrect when it is ultimately presented.

[0006] Accordingly, there is a need for AR presentation techniques that maintain a stable, consistent content position in a see-through display despite last-moment changes in display pose and despite latency in generating and delivering AR content. The present invention fulfdls this need among others.SUMMARY OF INVENTION

[0007] The following presents a simplified summary of the invention in order to provide a basic understanding of some aspects of the invention. This summary is not an extensive overview of the invention. It is not intended to identify key or critical elements of the invention or to delineate the scope of the invention. Rather, the summary is presented as a prelude to the more detailed description that is presented later.

[0008] In various embodiments, a computer in a vehicle receives AR content that has been generated based on an initial position (e.g., pose) of a see-through display. After receiving the AR content and before presenting it to the operator, the computer receives an updated position of the see-through display. The computer compares the initial position to the updated position to determine a change in position, and then presents the AR content on the see-through display with the displayed AR content adjusted based at least in part on the change in position. In this way, the AR content may be stabilized in the operator’s perceived field of view, even when the display has moved since the content was generated.

[0009] In some embodiments, the vehicle is an aircraft (including a high-performance aircraft such as a fighter jet). In some embodiments, the see-through display is worn by an operator of the vehicle and may be integrated into a helmet.

[0010] In some embodiments, the updated position of the see-through display is determined using a display position detector. By way of example, the display position detector may include an event camera positioned in the vehicle to monitor helmet motion and / or markings on the helmet. The event camera may provide high-temporal-resolution measurements thatenable accurate tracking of last-moment head movements caused by vibration, turbulence, g-force, and / or intentional head movement.

[0011] In some embodiments, the AR content includes predictive AR content based on motion data of the vehicle. For example, AR content may be generated for a predicted future time that is at least as long as an end-to-end latency associated with generating and delivering the AR content for display. In some embodiments, the AR content includes a time-to-present value, and the time-to-present may be selected to be at least as long as the latency. The AR content may be buffered until a presentation time, and then adjusted immediately prior to display based on the updated position of the see-through display.

[0012] In some embodiments, the AR content includes virtual content at a content position within an initial field of view of the see-through display. When the display moves and an updated field of view is defined, the AR content may be adjusted such that the content position remains the same relative to the operator’s perceived scene (e.g., by shifting or otherwise transforming one or more rendered video frames). In some embodiments, the AR content is a three-dimensional representation of the operator’s field of view.

[0013] In some embodiments, initial position data 172 defining the initial position (and thus an initial field of view) is transmitted from the vehicle to a remote computer 160 over a network 162, and AR content 170 is received at the vehicle from the remote computer. The remote computer may generate or assist in generating the AR content using an AR content generator 164 and the initial position data and other motion-related data, while the vehicle computer performs last-moment stabilization adjustments based on the updated position data.

[0014] In one independent aspect, a method is provided for presenting AR content stabilized on a see-through display. The method includes receiving, at a computer in a vehicle, AR content generated based on an initial position of the see-through display; before displaying the AR content, receiving an updated position of the see-through display; comparing the initial and updated positions to determine a change in position; and displaying the AR content with the displayed AR content adjusted based at least in part on the change in position.

[0015] In another independent aspect, a system is provided that includes a see-through display and one or more computing devices (for example, in a vehicle) configured to receive AR content generated based on an initial display position, obtain an updated display position before presentation, determine a change in display position, and present the AR content with a position adjustment so that the content is stabilized with respect to the surrounding environment.

[0016] In another independent aspect, a non-transitory computer-readable medium (or other software product) is provided that stores instructions which, when executed by one or more processors, cause performance of operations including receiving AR content generated based on an initial display position, obtaining an updated display position before presentation, determining a change in display position, and displaying the AR content with an adjustment based at least in part on the change in display position.

[0017] The present invention offer a number of advantages. The techniques disclosed herein can reduce perceived jitter and misalignment of AR content caused by last-moment display motion. By enabling AR content to be generated early enough to accommodate latency and then stabilized using updated pose information, embodiments can improve operator situational awareness, reduce distraction, and improve safety in high-dynamics environments such as aircraft cockpits.

[0018] By correcting content placement at or near the last available moment before presentation, embodiments can provide stable and accurately aligned AR content even when the environment exhibits unpredictable vibration and rapid operator motion.

[0019] By maintaining alignment, embodiments can help the operator perceive AR content as anchored at an intended location in the surrounding environment (e.g., a geospatial location or target) rather than drifting within the display.

[0020] By combining predictive rendering (to absorb latency) with last-moment pose correction (to absorb head shake), embodiments can provide a seamless and responsive AR experience in high-dynamics scenarios.BRIEF DESCRIPTION OF FIGURES

[0021] FIG. l is a schematic of one embodiment of the system of the present invention.

[0022] FIG. 2 illustrates an example process flow for correcting a position of AR content within a display field of view based on predicted conditions and an updated display position obtained before presentation.

[0023] FIG. 3 illustrates an example of stabilized AR content relative to a vibrating display field of view.

[0024] FIG. 4 illustrates an example timeline from content generation to content presentation, including buffering and last-moment field-of-view adjustment.DETAILED DESCRIPTION

[0025] In the following paragraphs, the present invention will be described in detail by way of example with reference to the accompanying drawings. The term “embodiment” as used herein refers to an example, instance, or illustration. A feature described in connection with an embodiment may be combined with features of other embodiments. The disclosed structures and methods are not limited to the specific embodiments described, and a given embodiment does not necessarily include all features described herein. In addition, embodiments or methods do not necessarily include the referenced feature(s).

[0026] Embodiments disclosed herein stabilize AR content presented on a see-through display in a dynamic environment by applying a last-moment adjustment to content placement based on an updated display position obtained after the AR content is received (and, in some embodiments, after it is rendered into one or more video frames). In practical terms, content may be generated or received early enough to accommodate pipeline latency, and then repositioned immediately prior to display using a refreshed estimate of the display pose.

[0027] As used herein, a “see-through display” includes, without limitation, a head-worn display (HWD), helmet-mounted display (HMD), waveguide-based combiner, optical combiner, or other display system that permits the user to view the physical environment while also viewing virtual content. A “position” of the display may include one or more of a location, an orientation, and / or a full six degree-of-freedom (6-DOF) pose in a coordinate system associated with the vehicle and / or environment. A “field of view” refers to a portion of space that is visible through the see-through display and may be modeled as a frustum or other projection geometry for rendering and content placement.

[0028] Referring to Fig. 1, one embodiment of the system 100 of the present invention is shown. In this embodiment, the system 100 presents augmented reality (AR) content stabilized on a see-through display and comprises: (a) a see-through display 152; (b) one or more processors 122 of a computer 120 in a vehicle 110; and (c) a memory 124 storing instructions that, when executed by the one or more processors, cause the one or more processors to: (i) receive, at the computer in the vehicle, AR content, said AR content being generated based on an initial position of said see-through display; (ii) following reception of said AR content and beforedisplaying AR content, receive an updated position of said see-through display; (iii) compare said initial position to said updated position to determine a change in position; and (iv) display said AR content on said see-through display, said displayed AR content being adjusted based at least in part on said change in position.

[0029] The computer 120 (e.g., one or more processors with associated memory) in the vehicle receives AR content, receives updated display position data, determines a change in display position, and presents the AR content with an adjustment based on the change in position. In some embodiments, the vehicle computer includes a rendering pipeline that can render or composite AR content into video frames suitable for the see-through display. The vehicle computer may also include a stabilization module that computes a transform between an initial display pose (used when the AR content was generated) and an updated display pose (measured near display time), and applies that transform to the AR content (or to rendered frames) so that content appears stable. In some embodiments, the vehicle computer is an avionics computer, mission computer, or dedicated AR processing unit. In other embodiments, at least part of the processing is performed by a wearable computer module associated with the helmet / display, by a separate graphics processing unit (GPU), and / or by a distributed set of processors within the vehicle.

[0030] The see-through display 152 presents virtual content aligned with the operator’s real-world view while permitting the operator to view the physical environment. The see-through display may be worn by the operator and may be integrated into a helmet 150. In some embodiments, the see-through display presents AR content at a display refresh rate (e.g., 60 Hz, 90 Hz, 120 Hz, or higher). The display may receive video frames and / or rendering commands from the vehicle computer, and may present content using a calibrated optical model that maps pixels to rays in the operator’s view. Although certain examples herein describe aircraft and helmet-mounted displays, the same stabilization concepts may be used with other vehicles (e.g., helicopters, unmanned aerial vehicles, ground vehicles, marine vehicles) and with other see-through display form factors (e.g., goggles, glasses, head-up displays, or fixed combiners).

[0031] A display position detector 140 determines an updated position of the see-through display after AR content is received and before the AR content is displayed, enabling last-moment stabilization adjustment. Tn some embodiments, the display position detector provides updated pose measurements at a rate sufficient to capture rapid head movements and vibration-induced motion. The updated pose may be represented as a quaternion and translation vector, Euler angles and translation, a homography, an optical flow field, and / or another representation suitable for computing an adjustment for content placement. With reference to FIG. 1, in one embodiment the display position detector comprises an event camera mounted within the cockpit or airframe and oriented to image a pilot’s helmet. The helmet may include passive reflective markers and / or active markers (e.g., LEDs, OLEDs) arranged in a known spatial configuration. By detecting changes in brightness associated with marker motion, the event camera may provide high-temporal-resolution measurements of helmet pose.

[0032] In some embodiments, the event camera is an imaging sensor that outputs an asynchronous stream of “events” corresponding to pixel-level changes in brightness rather than a sequence of full image frames captured at fixed intervals. Because event data is generated upon change, the event camera can provide very high temporal resolution and can be well -suited for capturing rapid motion and vibration. In some embodiments, the event camera also provides a high dynamic range, which can be advantageous in cockpits where sunlight and shadow can change quickly. The event stream may be processed to estimate marker trajectories, helmet pose, and / or line-of-sight changes at fine time scales.

[0033] In some embodiments, the event camera is configured to focus on a subset of the visual spectrum (e.g., infrared) and may be used with an illumination system that enhances marker visibility. Marker light may be modulated (e.g., strobed) to improve detectability and reduce unrelated event data. Event data may be processed using feature tracking and pose estimation (e.g., PnP -based estimation, filtering, and / or sensor fusion).

[0034] In other embodiments, the display position detector includes one or more of an inertial measurement unit (IMU) on the helmet, a gyroscope, an accelerometer, a magnetometer, lidar, radar, structured light, a frame-based camera, and / or a fusion of multiple sensors.

[0035] AR content is received at the vehicle computer, where the AR content was generated based on an initial position (pose) of the see-through display. In some embodiments, when AR content is generated, an initial display pose is captured and associated with the content(e g., as metadata). The initial pose may define an initial FOV and may determine where virtual content is placed in an initial coordinate system. The AR content may include one or more rendered frames, one or more 3D models with associated placement instructions, symbology definitions, and / or other data that can be presented in the display.

[0036] In some embodiments, the initial display pose is determined by an event camera or other position detector. In other embodiments, the initial display pose is determined by a combination of helmet IMU data 132 and cockpit-mounted sensing 130. In still other embodiments, the initial display pose is derived from a previously estimated pose and / or a filtered pose estimate.

[0037] Predictive rendering allows AR content to be generated sufficiently early to compensate for end-to-end latency, while still enabling stable placement at the intended presentation time. With reference to FIGS. 2 and 4, motion data of the vehicle (e.g., position, speed, attitude, pose, and / or navigation data) may be used to predict a future vehicle state at a predetermined future time. AR content may be generated for that future time, and may include a time-to-present value indicating when the content should be displayed. In some embodiments, the predetermined future time and / or time-to-present are selected to be at least as long as the measured or estimated end-to-end latency of the content pipeline.

[0038] By way of example, the process may begin by receiving telemetry and pose data from the vehicle, including at least one of vehicle position, vehicle orientation, speed, acceleration, attitude, and / or past pose estimates for the display over a period of time sufficient to establish a trend. The system may extrapolate this data to predict a future condition (e.g., 25 ms to 50 ms ahead of a current time) that corresponds to an intended presentation time. A content generation system (e.g., a game engine or mission graphics engine) may then generate AR content for the predicted future condition so that the content is geometrically correct from the predicted viewpoint and so that the content placement corresponds to a predicted line of sight from the operator’s predicted eye position to an intended geospatial location for the content.

[0039] In some embodiments, predictive content is generated for multiple candidate future times and buffered, and a best-matching frame is selected based on the actual time-to-present. In some embodiments, a remote computer generates predictive AR content usingtelemetry data transmitted from the vehicle. In other embodiments, predictive content is generated entirely onboard the vehicle.

[0040] In some embodiments, generated AR content is stored in a buffer (e.g., a frame buffer or ring buffer) together with metadata such as an initial display pose, a predicted presentation time, and / or a time-to-present interval. The buffer may allow the system to generate content early (to absorb latency) while still presenting content at a precise intended time. In some embodiments, the system monitors measured end-to-end latency (e.g., via timestamps or acknowledgements) and dynamically adjusts the prediction horizon and / or time-to-present to maintain synchronization with the display refresh schedule.

[0041] After AR content is received and before it is displayed, an updated display position is received and compared to the initial position to determine a change in position. In some embodiments, the change in position is computed as a relative transform between an initial 6-DOF pose and an updated 6-DOF pose. The change may be decomposed into rotational and translational components, and may be converted into an image-space adjustment (e.g., a pixel shift, warp, homography, or reprojection) applicable to the AR content.

[0042] In some embodiments, the change in position is computed at least in part in the coordinate system of the see-through display (e.g., a display-centered coordinate frame), and then converted into a display-space mapping. For example, a small rotation of the helmet about a yaw or pitch axis may correspond to a pixel shift of symbology in an opposite direction to maintain a stable world-referenced appearance. In some embodiments, the stabilization module maintains calibration parameters (e.g., camera intrinsics, display optical model parameters, marker geometry, and / or eye-relief estimates) so that pose deltas can be converted into accurate image-space adjustments.

[0043] In some embodiments, the difference between the initial and updated positions results from vibration, turbulence, g-force, and / or intentional head movement. In some embodiments, the updated pose is measured within a short time window prior to presentation (e.g., within 10 milliseconds, 5 milliseconds, or 1 millisecond of the time-to-present), so that last-moment movement is captured.

[0044] The AR content is displayed on the see-through display in an adjusted manner so that the virtual content remains stable despite changes in display position. In some embodiments, displaying includes rendering AR content into video frames using a computer in the vehicle and then providing the frames to the see-through display for presentation. The adjustment may be performed by shifting one or more rendered video frames to correspond to the change in display position. In other embodiments, the adjustment is performed by updating the pose used in a final-stage render or reprojection step prior to presentation.

[0045] In some embodiments, the adjustment is applied to a rendered frame by shifting pixels (e.g., translating the frame in x and y) according to the computed pose delta. In some embodiments, a small rotational delta is approximated by a 2D translation for near-center content, while in other embodiments a full warp (e.g., a homography or per-pixel reprojection) is applied to compensate for rotation. To reduce edge artifacts, the system may render with an overscan margin (i.e., render slightly larger than the displayed viewport) so that subsequent shifts or warps do not introduce empty regions. In some embodiments, the adjustment is applied only to one or more overlay layers (e.g., symbology), while the underlying real-world view remains unchanged.

[0046] The adjustment may include any one or more of shifting, rotating, warping, resampling, reprojection of 3D geometry, or compositing the AR content so that a content position appears fixed relative to the physical world. In some embodiments, the AR content includes a 3D representation of the operator’s FOV, and the adjustment corresponds to reprojection of the 3D content from the initial pose to the updated pose.

[0047] A remote computer may assist in generating AR content while the vehicle performs last-moment stabilization. In some embodiments, initial position data defining the initial display position (and thus an initial FOV) is obtained in the vehicle and transmitted to a remote computer remote from the vehicle. The remote computer may generate AR content based on the initial position data and other information (e.g., vehicle telemetry and / or mission data) and transmit the AR content back to the vehicle. The vehicle then receives an updated display position, computes a change in position, and adjusts the received AR content prior to display.

[0048] In some embodiments, the transmitted initial position data includes a timestamp and / or a predicted presentation time so that the remote computer can generate content for a specific future moment. The remote computer may return AR content together with metadata such as the initial pose used for generation, the time-to-present, and / or a sequence number identifying a frame. In some embodiments, the vehicle buffers multiple received frames and selects a frame whose time-to-present best matches the current presentation schedule. In some embodiments, communications between the vehicle and the remote computer are secured and may include error detection, retransmission, and / or graceful degradation strategies to maintain safe operation.

[0049] In some embodiments, the remote computer is a ground system communicating with the vehicle via a wireless link. In other embodiments, the remote computer is a separate onboard processing subsystem. In some embodiments, the vehicle may continue to generate fallback AR content locally if the remote link is degraded, and may apply the same stabilization techniques to locally generated content.

[0050] Virtual content may be associated with a content position within an initial FOV and adjusted so that the content position is the same in an updated FOV.

[0051] In some embodiments, virtual content is defined in a world coordinate system or vehicle coordinate system, and the display pose defines a mapping to image coordinates. When the display pose changes, the mapping is updated so that the same world-referenced content projects to the same perceived location relative to the operator’s line of sight. In other embodiments, the AR content includes metadata identifying a content position in the initial FOV, and the stabilization module transforms the content position to a corresponding location in the updated FOV.

[0052] An AR display FOV may shift due to display motion, yet the content placement position can be shifted in an opposite direction to compensate, thereby maintaining a stable perceived content position. In some embodiments, the content position corresponds to a geospatial location or target point in the environment, and stabilization preserves the line of sight from the operator’s eyes to that location.

[0053] Fig. 2 illustrates a computer process flow diagram for correcting the position of AR content within a AR display field of view (FOV) due to last moment display movement 200, according to the principles of the present inventions. The AR content may be positioned within the FOV of the pilot’s AR display system at a position that will cause the pilot to associate the position with an actual geospatial position. This may involve receiving data from or about an aircraft indicting the aircraft’s position, speed, pose, location, altitude and the position of the pilot’s helmet 202. This data, gathered over time, can be used in an extrapolation equation to project the vehicle’s conditions at a future point in time 204. For example, the latency in communications between a content frame and / or rendering system and presenting the content to the pilot’s AR display may be 25ms, so the system may estimate the future data to be presented 50ms from the frame generation time such that the content can be presented at the intended time with some time to spare in case there are communication or processing errors that slow things down. The framing of the content, rendering of the content or other preparation step for the content may then be completed based on the future estimations 206. The AR content (e.g., video frame) may be buffered until the system indicates it is time to present the content to the AR display 208. As the time to present approaches (e.g., 10ms before presentation time) the head / helmet position detection system may be consulted to understand a refreshed position of the helmet, which may be based on G-forces, intentional head movement, shaking, etc. 210 to calculate a new FOV position upon which to position the content 212. Then, when it is time to present the AR content / video frame (e.g., when the current time is near or at the 50ms future point in time), the content can be presented at the new point in the FOV to compensate for the last moment movement 214. In embodiments, the content may not be redefined or changed before presenting at step 214. In other embodiments, the content may be redefined or changed before the presenting step at 214.

[0054] In some embodiments, the reassessment of display position and the repositioning of content (e.g., the operations corresponding to steps 210-214 of FIG. 2) are performed at or near the last available moment before the intended presentation time. By way of example, the updated pose may be sampled and the frame shift / warp may be computed beginning about 10 milliseconds, 5 milliseconds, 1 millisecond, or microseconds before a frame is displayed, subject to computational and communications constraints.

[0055] Fig. 3 illustrates a stabilized AR content position within a vibrating FOV of a AR display 300 in accordance with the principles of the present inventions. The one AR display’s FOV is illustrated in three positions 302, 304, and 306 to demonstrate a shaking display. As can be seen, the FOV can bounce around as the pilot’s head and / or helmet shakes. Content position 308 illustrates a stabilized position regardless of the FOV shake. The stabilized position is maintained by moving the content placement position in a direction and to a position within the FOV to compensate for the display shake (e.g., as measured through a helmet position tracker). The result provides stabilized content with respect to the line of sight describing the point of view from the pilot’s eyes to the geospatial location where the AR content is to be positioned within the surrounding environment.

[0056] Fig. 4 illustrates a frame-by-frame timeline from content generation to content presentation in a AR display 400. Timelines 416, 417 and 418 illustrate three sequential frames of video and how the frames are positioned within the FOV of an AR display to compensate for shaking or other movement of the AR display. The total time allotted for the process from 402 to 414 is designed to manage latency caused by communications and processing between and within separate computer systems and the AR display. For example, if the computer system includes wireless transmission of the content from ground to air it may take 50ms to transmit the content from end to end. If this is the situation, the process 400 may be at least 50ms prior to the estimated time for presentation of the content in the display. Tn other words, if it takes 50ms to get the content generated and presented, steps 404 to 414 should occur within the 50ms. Frame 416 starts first, followed by frame 417, and then 418. The frame speed is aligned with the desired refresh rate of the AR display. For example, if the desired refresh rate is 120Hz, the speed of the frame generation with be done at or around 120Hz. The 120Hz is the frame rate and the 50ms processing time represents the time shift to get the content presented at the correct position within the FOV based on last moment movements of the display.

[0057] The process 400 may begin with receiving data 402 from an aircraft in which a pilot is wearing a helmet with an integrated AR display or other form of display, either worn or mounted within the cockpit. The data, as described herein elsewhere, may be indicative of the aircraft’s position, speed, pose along with the pilot’s head position over a period of time (e g., sufficient data to establish a trend). This data may be used to predict the same aircraft and pilothead position for a future moment in time 404. The future moment in time may be fixed or dynamic and is intended to be at least as long as or longer than the latency of the computer communication and processing system in generating the AR content and presenting it to the pilot. A gaming engine, or other content generation system, may then generate the AR content based on the predicted future condition data such that it appears geometrically correct from the predicted future positions and it has an FOV position that is based on a line of sight from the pilot’s future location, plane pose and head position to the geospatial position representing the geospatial position of the AR content 406. The generated AR may be buffered 408 such that the timing of presenting the AR content is as intended. Then, just prior to the presentation of the AR content to the pilot’s AR display at step 414, a head tracking system (e.g., event camera) reassesses the pilot’s head position as a proxy for the AR display position and the pilot’s line of sight to the intended geospatial location for the AR content 410. And, the AR content is then repositioned at a new location within the AR display FOV to compensate for the new display position 412. The steps of 410 to 414 or 412 to 414 may be done at or near the last available moment before the future time arrives, which is mainly limited by communication lag and computing lag. So, these steps may be started at a point 10ms, 5ms, 1ms, or microseconds in advance of the presenting step 414, for example.

[0058] These and other advantages may be realized and attained by means of the instruments and combinations particularly pointed out in the appended claims. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention as claimed. Accordingly, the invention should be understood to cover the full scope of equivalents to which such claims are entitled.

Claims

1. WHAT IS CLAIMED IS:

1. Method for presenting augmented reality (AR) content stabilized on a see-through display, said method comprising:a. receiving, at a computer in a vehicle, AR content, said AR content being generated based on an initial position of said see-through display; b. following reception of said AR content and before displaying AR content, receiving an updated position of said see-through display;c. comparing said initial position to said updated position to determine a change in position; andd. displaying said AR content on said see-through display, said displayed AR content being adjusted based at least in part on said change in position.

2. The method of claim 1, wherein said vehicle is an aircraft.

3. The method of claim 1, wherein said see-through display is worn by an operator of said vehicle.

4. The method of claim 1, wherein said see-through display is integrated into a helmet.

5. The method of claim 1, wherein said updated position of said see-through display is determined using a display position detector.

6. The method of claim 5, wherein said display position detector is an event camera in said vehicle.

7. The method of claim 6, wherein said initial display position is determined by said event camera.

8. The method of claim 1, wherein said AR content comprises predictive AR content based on motion data of said vehicle.

9. The method of claim 8, wherein said predictive AR content is predictive of a predetermined future time at least as long as an end-to-end latency associated with generating and delivering said AR content for display.

10. The method of claim 9, wherein said AR content comprises a time to present.

11. The method of claim 10, wherein said time to present is at least as long as said latency.

12. The method of claim 1, wherein said AR content is a 3D representation of a field of view (FOV) of an operator13. The method of claim 6, wherein said difference between said first and second positions is a result of at least one of g-force, vibration, turbulence, or intentional head movement.

14. The method of claim 1, wherein said displaying comprises at least rendering said AR content into video frames using a computer in said vehicle.

15. The method of claim 14, wherein said displayed AR content is adjusted by shifting one or more video frames to correspond to said change of position of said see-through display.

16. The method of claim 1, further comprising:obtaining initial position data of said initial position of said see-through display, said initial position defining an initial field of view of said see-through display; andtransmitting said initial position data to a remote computer remote from said vehicle.

17. The method of claim 1, wherein said initial position defines an initial field of view of said see-through display, and said updated position defines an updated field of view of said see-through display.

18. The method of claim 17, wherein said AR content has virtual content in a content position in said initial field of view.

19. The method of claim 18, wherein said AR content is adjusted such that said content position is the same in said initial field of view and in said updated field of view.

20. The method of claim 1, wherein said AR content is received from said remote computer.

21. A system for presenting augmented reality (AR) content stabilized on a see-through display, comprising:a see-through display;one or more processors of a computer in a vehicle; anda memory storing instructions that, when executed by the one or more processors, cause the one or more processors to:receive, at the computer in the vehicle, AR content, said AR content being generated based on an initial position of said see-through display;following reception of said AR content and before displaying AR content, receive an updated position of said see-through display;compare said initial position to said updated position to determine a change in position; anddisplay said AR content on said see-through display, said displayed AR content being adjusted based at least in part on said change in position.

22. The system of claim 21, wherein said see-through display is integrated into a helmet.

23. The system of claim 21, wherein said updated position of said see-through display is determined using a display position detector.

24. The system of claim 25, wherein said display position detector is an event camera in said vehicle.

25. The system of claim 21, wherein said AR content comprises predictive AR content based on motion data of said vehicle.

26. The system of claim 28, wherein said predictive AR content is predictive of a predetermined future time at least as long as an end-to-end latency associated with generating and delivering said AR content for display.

27. The system of claim 21, wherein said displaying comprises at least rendering said AR content into video frames using a computer in said vehicle.

28. The system of claim 27, wherein said displayed AR content is adjusted by shifting one or more video frames to correspond to said change of position of said see-through display.

29. The system of claim 21, wherein said AR content is received from said remote computer.

30. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a computer in a vehicle, cause the one or more processors to perform operations for presenting augmented reality (AR) content stabilized on a see-through display, the operations comprising: receiving, at the computer in the vehicle, AR content, said AR content being generated based on an initial position of said see-through display; following reception of said AR content and before displaying AR content, receiving an updated position of said see-through display; comparing said initial position to said updated position to determine a change in position; and displaying said AR content on said see-through display, said displayed AR content being adjusted based at least in part on said change in position.

Images