Improving spatial localization accuracy in neural networks

Abstract

Systems, methods, and devices that relate to localized normalized exponential functions for addressing multi-peak heatmaps in pose estimation are disclosed. In one example aspect, the method includes receiving a heatmap including intensity values at spatial positions. The method includes identifying a peak position having a maximum intensity value using an ArgMax operation. The method generates a filtered heatmap by setting intensity values beyond a predetermined distance threshold from the peak position to zero while retaining values within the threshold. The method applies a SoftArgMax operation to the filtered heatmap to compute a weighted average of spatial positions based on normalized exponential values. The method outputs coordinates corresponding to the weighted average as a localized position estimate for pose estimation applications.

Description

PATENTIMPROVING SPATIAL LOCALIZATION ACCURACY IN NEURAL NETWORKSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 713,503, titled LOCALIZED NORMALIZED EXPONENTIAL FUNCTION FOR ADDRESSING MULTI-PEAK HEATMAPS USED FOR POSE ESTIMATION, filed October 29, 2024, which is hereby incorporated by reference in its entirety.FIELD OF INVENTION

[0002] Various embodiments concern computer programs and associated computer-implemented techniques for improving spatial localization accuracy in neural networks through localized normalized exponential functions that address multi-peak heatmaps in pose estimation applications.BACKGROUND

[0003] Neural networks have become fundamental tools in computer vision applications, particularly for tasks involving spatial localization and pose estimation. These systems rely on gradient-based optimization methods during training, which require all operations within the network to be differentiable to enable proper backpropagation of error signals through the network layers.

[0004] Traditional ArgMax operations, while effective at identifying peak locations in data arrays, present a challenge in neural network training due to their non- differentiable nature. The ArgMax function returns discrete indices corresponding to maximum values, creating discontinuous jumps that prevent the computation of meaningful gradients. This limitation makes ArgMax operations unsuitable for end-to- end training of neural networks where smooth gradient flow is required for parameter optimization.

[0005] To address this limitation, SoftArgMax functions have been developed as differentiable approximations of ArgMax operations. SoftArgMax transforms input vectors into probability distributions using exponential functions and computes weighted averages of spatial positions. This approach maintains differentiability while providing continuous outputs that approximate the behavior of ArgMax operations, making them suitable for integration into trainable neural network architectures.1184066260 1PATENT

[0006] However, SoftArgMax operations face challenges when applied to heatmaps containing multiple peaks or significant noise artifacts. In such scenarios, the weighted averaging nature of SoftArgMax can blend information from multiple distinct peaks, resulting in output coordinates that fall between actual peak locations rather than accurately identifying any single peak. This blending effect can reduce localization precision and introduce ambiguity in applications where precise spatial identification is required.

[0007] The presence of multiple peaks in heatmaps commonly occurs in pose estimation applications, particularly when neural networks exhibit uncertainty about spatial distinctions or when input data contains ambiguous visual information. For example, left-right joint ambiguities can produce dual peaks in heatmaps, leading SoftArgMax operations to output coordinates positioned between the two peaks rather than selecting the most probable location. Additionally, noise and spurious artifacts in heatmap data can introduce false peaks that further compromise the accuracy of SoftArg Max-based localization.

[0008] These challenges become particularly pronounced in applications requiring high spatial precision, such as medical imaging analysis, autonomous vehicle navigation, augmented reality systems, and robotic manipulation tasks. In these domains, inaccurate localization can have significant consequences for system performance and reliability. Current approaches to mitigate these issues often involve switching between different operations during training and inference phases, which can create inconsistencies in the signals received by neural networks and potentially degrade overall system performance.BRIEF DESCRIPTION OF FIGURES

[0009] Figure 1 illustrates a network environment that includes a pose estimation platform that is executed by a computing device.

[0010] Figure 2 illustrates an example of a computing device that is able to implement a platform designed to facilitate pose estimation.

[0011] Figure 3 illustrates a pose estimation diagram showing skeletal joint detection and tracking, according to aspects of the present disclosure.

[0012] Figure 4 illustrates a flow for processing heatmap data through multiple computational steps, according to aspects of the present disclosure.2184066260 1PATENT

[0013] Figure 5 depicts a graph showing signal intensity variations with SoftArgMax operations, according to aspects of the present disclosure.

[0014] Figure 6 includes a flow diagram of a process for pose estimation.

[0015] Figure 7 is an example of a machine learning model.

[0016] Figure 8 is a block diagram illustrating an example of a processing system that can perform at least some operations described herein.

[0017] Various features of the technology will become more apparent to those skilled in the art from a study of the Detailed Description in conjunction with the drawings. In the drawings, embodiments are illustrated by way of example and not limitation for the purpose of illustration. Those skilled in the art will recognize that alternative embodiments may be employed without departing from the principles of the present disclosure. Accordingly, while specific embodiments are shown in the drawings, the technology is amenable to various modifications.DETAILED DESCRIPTION

[0018] The approach disclosed herein is designed to address the aforementioned challenges in spatial localization accuracy within neural networks used for pose estimation applications. The approach involves a computational framework that addresses the fundamental problem of multi-peak heatmaps that can compromise the accuracy of traditional SoftArgMax operations. In particular, the framework identifies the most prominent peak in a heatmap and applies localized filtering to eliminate interference from distant peaks and noise artifacts. This targeted approach enables neural networks to maintain differentiability while achieving more precise spatial localization than conventional methods.

[0019] To enable neural networks to overcome the limitations of standard SoftArgMax operations, the localized normalized exponential function applies a selective filtering mechanism to heatmap data. The framework includes a peak identification stage using ArgMax operations, followed by distance-based filtering that retains only intensity values within a predetermined threshold of the identified peak. By isolating the primary peak region and eliminating distant values, the approach ensures that subsequent SoftArgMax computations focus on the most relevant spatial information. This localized processing transforms the multi-peak problem into a singlepeak scenario, enabling more accurate coordinate prediction for joint localization tasks.3184066260 1PATENT

[0020] The computational approach outlined above can effectively address current challenges in pose estimation accuracy. The localized normalized exponential function can process heatmaps containing multiple peaks or ambiguous spatial information, significantly reducing the blending effects that compromise traditional SoftArgMax operations. This method is more precise and computationally efficient than approaches that attempt to handle multi-peak scenarios through post-processing or training-inference inconsistencies. The framework can ensure that the spatial coordinates generated correspond to actual peak locations rather than interpolated positions between multiple peaks, facilitating more accurate joint localization and pose tracking.

[0021] The localized normalized exponential function can include adaptive threshold mechanisms that adjust the predetermined distance threshold based on feedback data and localization accuracy metrics. This adaptive capability can provide filtering parameters that optimize performance for specific pose estimation tasks and environmental conditions. The adaptive structure can impact both training and inference processes by allowing for more specialized and context-aware localization. The threshold adjustment mechanism can be continuously refined based on ground truth comparisons, enabling the system to learn optimal filtering parameters for different types of pose estimation scenarios.

[0022] The localized normalized exponential function can provide computational efficiency by applying selective processing only to relevant spatial regions. This means that the SoftArgMax computation, which can be computationally intensive for large heatmaps, can be applied only to the filtered subset of intensity values within the predetermined threshold. This selective application can ensure that computational resources are used effectively, reducing processing time while maintaining or improving localization accuracy. Furthermore, this approach can enhance the reliability and precision of pose estimation by eliminating the influence of spurious peaks and noise artifacts that can mislead traditional SoftArgMax operations.

[0023] Another advantage of the localized approach is the ability to handle various types of heatmap data across different dimensionalities. This flexible framework can optimize performance for one-dimensional, two-dimensional, or higher-dimensional heatmaps used in different pose estimation applications. For example, the method can be applied to two-dimensional joint probability distributions for human pose estimation, where it can provide more accurate anatomical joint coordinates within image frames.4184066260 1PATENTThe separation of peak identification and localized filtering can lead to more robust and scalable solutions capable of handling diverse pose estimation scenarios and varying levels of spatial complexity.

[0024] The framework can further facilitate modularity and adaptability within pose estimation systems. By isolating the peak identification and filtering processes into distinct computational stages, developers can optimize individual components without affecting the entire pipeline. This modular approach can simplify the integration of the localized normalized exponential function into existing neural network architectures, ensuring that pose estimation platforms can benefit from improved spatial localization without requiring comprehensive system redesigns. Additionally, the clear separation enhances the robustness of the overall system, as issues within one processing stage can be addressed independently without causing widespread disruptions to the pose estimation pipeline.Terminology

[0025] References in the present disclosure to “an embodiment” or “some embodiments” mean that the feature, function, structure, or characteristic being described is included in at least one embodiment. Occurrences of such phrases do not necessarily refer to the same embodiment, nor are they necessarily referring to alternative embodiments that are mutually exclusive of one another.

[0026] Unless the context clearly requires otherwise, the terms “comprise,” “comprising,” and “comprised of” are to be construed in an inclusive sense rather than an exclusive or exhaustive sense — that is, in the sense of “including but not limited to.” The term “based on” is also to be construed in an inclusive sense. Thus, the term “based on” is intended to mean “based at least in part on.”

[0027] The terms “connected,” “coupled,” and variants thereof are intended to include any connection or coupling between two or more elements, either direct or indirect. The connection or coupling can be physical, logical, or a combination thereof. For example, elements can be electrically or communicatively coupled to one another despite not sharing a physical connection.

[0028] The term “module” can refer broadly to software, firmware, hardware, or combinations thereof. Modules are typically functional components that generate one or more outputs based on one or more inputs. A computer program can include or utilize one or more modules. For example, a computer program can utilize multiple5184066260 1PATENT modules that are responsible for completing different tasks, or a computer program can utilize a single module that is responsible for completing all tasks.

[0029] When used in reference to a list of multiple items, the word “or” is intended to cover all of the following interpretations: any of the items in the list, all of the items in the list, and any combination of items in the list.Overview of Pose Estimation Platform

[0030] Figure 1 illustrates a network environment 100 that includes a pose estimation platform 102 (or simply “pose estimation platform”) that is executed by a computing device 104. An individual (also referred to as a “user”) can interact with the pose estimation platform 102 via interfaces 106. For example, a user (e.g., a researcher or developer) is able to access an interface through which information (e.g., pose data, joint coordinates, etc.) is displayed. As another example, a user (e.g., a system operator) is able to access an interface through which various measures associated with pose estimation can be viewed. Some interfaces 106 can be designed to allow for the review of information acquired, derived, or produced by the pose estimation platform 102 (e.g., by researchers), while other interfaces 106 can be designed to facilitate information gathered from users (e.g., from system operators).

[0031] As shown in Figure 1 , the pose estimation platform 102 can reside in a network environment 100. Thus, the computing device 104 on which the pose estimation platform 102 resides can be connected to one or more networks 108A- 108B. Depending on its nature, the computing device 104 can be connected to a personal area network (PAN), local area network (LAN), wide area network (WAN), metropolitan area network (MAN), or cellular network. For example, if the computing device 104 is a computer server, then the computing device 104 can be accessible to users via respective mobile phones that are connected to the Internet via LANs. Data to be examined by the pose estimation platform 102 can be generated by the respective mobile phones or acquired by the respective mobile phones. Alternatively, the computing device 104 can be associated with, and accessible to, a user — in which case the computing device 104 can be connected to a server system 110 that is responsible for supporting the pose estimation platform 102. In such embodiments, the computing device 104 can be a mobile phone, tablet computer, or wearable computing device (e.g., a fitness tracker or watch), for example.

[0032] Additionally or alternatively, the computing device 104 can be connected to one or more other computing devices over a short-range wireless connectivity6184066260 1PATENT technology, such as Bluetooth®, Near Field Communication (NFC), Wi-Fi® Direct (also referred to as “Wi-Fi P2P”), and the like. As an example, the pose estimation platform 102 can be embodied as a mobile application that is executed by a mobile phone. In such embodiments, the mobile phone can be communicatively connected — via a wireless communication channel — to a source from which to acquire data. The source can be a camera, depth sensor, or another imaging device, for example. The data can alternatively be obtained from another computer program executing on the mobile phone. For example, the data can instead be acquired from another mobile application executing on the mobile phone or the operating system of the mobile phone.

[0033] The interfaces 106 can be accessible via a web browser, desktop application, mobile application, or another form of computer program. For example, a user is able to access interfaces through which information regarding pose estimation results can be provided, viewed, or altered via a mobile application executing on a mobile phone. Through these interfaces, the pose estimation platform 102 can provide queries, updates, or results to the user regarding pose detection and tracking.

[0034] Generally, the pose estimation platform 102 is executed — at least partially — by a cloud computing service operated by, for example, Amazon Web Services®, Google Cloud Platform™, or Microsoft Azure®. Thus, the computing device 104 can be representative of a computer server that is part of a server system 110. Often, the server system 1 10 includes multiple computer servers. These computer servers can include different types of data (e.g., pose models, joint coordinates, heatmaps, etc.), algorithms for processing data, models (e.g., neural networks, pose estimation models, etc.) for detecting poses or tracking movement, and other assets. Those skilled in the art will recognize that these data can also be distributed among the server system 1 10 and one or more computing devices. For example, some data that is input by, or related to, users can be stored on, and processed by, their own computing devices for security or privacy purposes. As a specific example, certain data that is captured by a user with their own device can remain on that computing device, though insights derived via analysis of the data can be transmitted external to that computing device (e.g., to the server system 1 10 for further consideration or analysis).

[0035] Components of the pose estimation platform 102 can also be hosted locally. That is, part of the pose estimation platform 102 can reside on the computing device7184066260 1PATENT used to access one of the interfaces 106. For example, the pose estimation platform 102 can be embodied as a mobile application executing on a mobile phone as mentioned above. Note, however, that the mobile application can be communicatively connected to the server system 1 10 on which other components of the pose estimation platform 102 are hosted.

[0036] Figure 2 illustrates an example of a computing device 200 that is able to implement a pose estimation platform 212. As shown in Figure 2, the computing device 200 can include a processor 202, memory 204, display mechanism 206, and communication module 208. Each of these components is discussed in greater detail below.

[0037] Those skilled in the art will recognize that different combinations of these components can be present depending on the nature of the computing device 200. For example, if the computing device 200 is a computer server that is part of a server system (e.g., server system 1 10 of Figure 1 ), then the computing device 200 may not include the display mechanism 206. Conversely, if the computing device 200 is a mobile phone, then the computing device 200 can include the display mechanism 206.

[0038] The processor 202 can have generic characteristics similar to general- purpose processors, or the processor 202 can be an application-specific integrated circuit (ASIC) that provides control functions to the computing device 200. As shown in Figure 2, the processor 202 can be coupled to all components of the computing device 200, either directly or indirectly, for communication purposes.

[0039] The memory 204 can be comprised of any suitable type of storage medium, such as static random-access memory (SRAM), dynamic random-access memory (DRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, or registers. In addition to storing instructions that can be executed by the processor 202, the memory 204 can also store data generated by the processor 202 (e.g., when executing the modules of the pose estimation platform 212). Note that the memory 204 is merely an abstract representation of a storage environment. The memory 204 can be comprised of actual integrated circuits (also called “chips”).

[0040] The display mechanism 206 can be any mechanism that is operable to visually convey information to a user. For example, the display mechanism 206 can be a panel that includes light-emitting diodes (LEDs), organic LEDs, liquid crystal elements, or electrophoretic elements. As further discussed below, outputs produced by the pose estimation platform 212 (e.g., through execution of its modules) can be8184066260 1PATENT posted to the display mechanism 206 for review by a user of the computing device 200. In some embodiments, the user can be an individual engaging with the pose estimation platform 212.

[0041] The communication module 208 can be responsible for managing communications external to the computing device 200. The communication module 208 can be wireless communication circuitry that is able to establish wireless communication channels with other computing devices. Examples of wireless communication circuitry include 2.4 gigahertz (GHz) and 5 GHz chipsets compatible with Institute of Electrical and Electronics Engineers (IEEE) 902.12 — also referred to as “Wi-Fi chipsets.” Alternatively, the communication module 208 can be representative of a chipset configured for Bluetooth, NFC, and the like. Some computing devices— like mobile phones, tablet computers, and the like— are able to wirelessly communicate via separate channels, while other computing devices — like watches and fitness trackers — tend to wirelessly communicate via a single channel. Accordingly, the communication module 208 can be one of multiple communication modules implemented in the computing device 200, or the communication module 208 can be the only communication module implemented in the computing device 200.

[0042] The nature, number, and type of communication channels established by the computing device 200 — and more specifically, the communication module 208 — can depend on (i) the sources from which data is received by the pose estimation platform 212 and (ii) the destinations to which data is transmitted by the pose estimation platform 212. Assume, for example, that the pose estimation platform 212 resides on a server. In such embodiments, the communication module 208 can communicate with sources 210A-210N external to the computing device 200 from which to obtain data. This data can include image data, video data, training data, information about various poses, skeletal information, joint coordinates, depth information, contextual information, or other relevant data. As mentioned above, data can be acquired by the pose estimation platform 212 from one of the sources 210A- 21 ON, derived by the pose estimation platform 212 (e.g., via analysis of sensor data, metadata, etc.), or simply provided by users (e.g., via a survey, a test, or other mechanism).

[0043] Note that the term “pose information” can also be used to refer to data that is ingested, acquired, or otherwise obtained by the pose estimation platform 212. The terms “pose information” and “pose data” can be used to refer to image data, skeletal9184066260 1PATENT information, joint coordinates, and other potential pose-related information associated with an individual (e.g., regarding body positions, movements, gestures, etc.). Other types of data can include information that is not directly related to pose estimation but can affect the system more generally. Examples include contextual information as well as information related to lighting conditions, camera parameters, environmental factors, and the like.

[0044] For convenience, the pose estimation platform 212 is referred to as a computer program that resides within the memory 204. However, the pose estimation platform 212 can be comprised of software, firmware, or hardware that is implemented in, or accessible to, the computing device 200. In accordance with embodiments described herein, the pose estimation platform 212 can include a processing module 214, a training module 216, a machine learning module 218, an assessment module 220, and a graphical user interface (GUI) module 222. These modules can be integral parts of the pose estimation platform 212, or they can be logically separate from the pose estimation platform 212 but operate “alongside” it.

[0045] The processing module 214 can process data that is obtained by the pose estimation platform 212 into a format that is suitable for the other modules. For example, the processing module 214 can apply operations to data acquired from the sources 210A-210N in preparation for analysis by the other modules of the pose estimation platform 212. For example, the processing module 214 can filter or alter the data such that the data can be more readily analyzed. As another example, the processing module 214 can parse different types of data (e.g., image data) or data provided via different mechanisms (e.g., data directly uploaded to pose estimation platform 212 and data obtained by pose estimation platform 212 from memory 204 or another computer program executing on the computing device 200) in order to temporally arrange these data, thereby ensuring that any insights gleaned through analysis of these data are temporally sensical. Such an approach can be helpful in ensuring that changes — for example, in pose data — are properly detected.

[0046] The pose estimation platform 212 utilizes information stored in different knowledge bases to gain insights into the poses and corresponding detection algorithms. These knowledge bases can be stored in the memory 204, or these knowledge bases can be external to the computing device 200 but accessible via the communication module 208. These knowledge bases can include pose models, joint configurations, skeletal structures, movement patterns, or other information. These10184066260 1PATENT knowledge bases can be constructed based on an analysis of data related to users, data provided by users, or other data.

[0047] The machine learning module 218 can apply a model suite to information associated with a user in order to detect poses and track movement. The model suite can include neural networks, supervised and unsupervised models, and learning engines. The training module 216 can be responsible for training these models. In some embodiments, the training module 216 can train models such as those discussed in relation to Figure 7.

[0048] The assessment module 220 can perform assessments of pose detection accuracy, apply evaluation algorithms, or perform other actions. In some embodiments, the assessment module 220 can utilize various data inputs, such as ground truth annotations, validation datasets, and sensor data, to evaluate the system’s performance. For example, the assessment module 220 can analyze the accuracy and consistency of pose detection, the precision of joint localization, and the system’s robustness to various conditions. By applying sophisticated assessment algorithms, the assessment module 220 can generate a comprehensive performance report that highlights the system’s strengths and areas needing improvement. Additionally, the assessment module 220 can identify patterns and trends in the pose detection results, providing valuable insights that can inform future model improvements. This detailed analysis can help tailor the algorithms and parameters used by the system, ensuring that the pose estimation platform remains effective and aligned with the evolving requirements. Furthermore, the assessment module 220 can trigger specific actions based on the evaluation results, including model adjustments, parameter tuning, or algorithm refinements, to keep the system performing optimally.

[0049] GUI module 222 can present a series of interfaces through an intuitive GUI aimed at displaying pose estimation results and relevant information about the detected poses, joint positions, and other pertinent factors. For example, the GUI module 222 can show users the detected skeletal structure, joint coordinates, confidence scores, and any recent changes in pose detection. The GUI module 222 can provide functionality to capture user feedback and annotations in various formats, such as text input, audio or video recordings, or other formats. As users interact with the pose estimation results, the GUI module 222 can capture and record their feedback in real time. In some embodiments, the GUI module 222 can collect user1 1184066260 1PATENT feedback, which can be used as reference feedback by the training module 216 for training machine learning models.Overview of Methods for Pose Estimation

[0050] Figure 3 illustrates a pose estimation diagram 300 showing skeletal joint detection and tracking, according to aspects of the present disclosure. The pose estimation diagram 300 depicts a human form in a standing position with bent knees, with several joint positions and estimations marked. For example, a left knee position 302 and a right knee position 304 are indicated on the figure. The pose estimation diagram 300 also shows a left knee estimation 306 and a right knee estimation 308, which represent predicted positions for these joints. This visual representation illustrates the challenges that can arise when neural networks exhibit uncertainty about spatial distinctions, particularly in cases where left-right joint ambiguities can produce dual peaks in heatmaps. The discrepancies between the actual joint positions and their corresponding estimations demonstrate how traditional pose estimation approaches using conventional operations can suffer from reduced accuracy when processing multi-peak heatmaps. In such scenarios, the weighted averaging nature of standard SoftArgMax functions can cause predicted joint coordinates to fall between actual anatomical landmarks rather than accurately identifying the true joint locations, leading to anatomically inconsistent pose representations and compromised tracking performance in applications requiring precise spatial localization.

[0051] The approach described herein addresses these challenges through a localized normalized exponential function that improves spatial localization accuracy in neural networks. In some implementations, the platform first receives a heatmap from a neural network that shows intensity values at different spatial positions. The heatmap can function as a probability map showing where joints are likely to be located. The system then identifies the peak position with the highest intensity using an ArgMax operation, which finds the most probable location. However, instead of applying traditional SoftArgMax operations to the entire heatmap (which can get confused by multiple peaks), the approach creates a filtered version by keeping only the intensity values within a certain distance of that peak position and setting everything else to zero. This filtering step eliminates distant peaks and noise that could interfere with accurate localization. Finally, the system applies a SoftArgMax operation to this filtered heatmap to compute a weighted average of spatial positions, which provides the final coordinates for joint localization. This localized approach enables12184066260 1PATENT more precise pose estimation by focusing computational resources on the most relevant spatial regions while maintaining the differentiability needed for neural network training.

[0052] In particular, the pose estimation platform can receive, via a computing interface, a heatmap indicating intensity values at spatial positions. In other implementations, the heatmap can be generated for a spatial localization application. A heatmap represents a data visualization technique where individual values contained in a matrix are represented as colors, with intensity values corresponding to the magnitude or probability of a particular feature at specific spatial coordinates. For example, in pose estimation applications, heatmaps can represent the probability distribution of joint locations within an image frame, where higher intensity values indicate greater likelihood of a joint being present at that position. The computing interface can include various input mechanisms such as application programming interfaces (APIs), network connections, file system interfaces, or direct memory access channels.

[0053] In some implementations, the heatmap includes a two-dimensional array representing joint probability distributions for human pose estimation. In particular, the localized position estimate corresponds to predicted coordinates of a specific anatomical joint within an image frame. Joint probability distributions represent the likelihood of finding particular anatomical features at specific pixel locations within an image, with higher probability values indicating greater confidence in joint presence. For example, a heatmap for a knee joint can show high probability values in the region where the knee is likely located, with probability values decreasing as distance from the actual joint position increases. The two-dimensional array structure enables efficient storage and processing of spatial probability information, with array indices corresponding to pixel coordinates and array values representing probability magnitudes. Anatomical joints can include major body landmarks such as shoulders, elbows, wrists, hips, knees, and ankles, each requiring separate heatmap processing for accurate localization. The predicted coordinates generated by the localized position estimate can be expressed in various coordinate systems including pixel coordinates relative to the image frame, normalized coordinates scaled to unit dimensions, or world coordinates calibrated to real-world measurements.

[0054] In some implementations, the heatmap is generated by a neural network for a pose estimation application. The neural network generating the heatmap can be13184066260 1PATENT implemented using various architectures including convolutional neural networks (CNNs), recurrent neural networks (RNNs), or transformer-based models specifically trained for spatial localization tasks. The neural network can be trained using supervised learning techniques with annotated training datasets containing ground truth joint positions, where the network learns to map input images to probability distributions represented as heatmaps through backpropagation optimization. During training, the network minimizes a loss function that measures the difference between predicted heatmaps and ground truth heatmaps, typically using mean squared error or cross-entropy loss functions. The training process involves feeding the network thousands of labeled images showing human poses with corresponding ground truth joint coordinates, enabling the network to learn feature representations that correlate visual patterns with spatial joint locations. The trained neural network generates heatmaps by processing input images through multiple layers of feature extraction and spatial reasoning, ultimately producing output tensors where each spatial location contains a probability value indicating the likelihood of a joint being present at that position. The neural network can use any of the techniques described in relation to Figure 7. Moreover, the heatmap can be represented as one-dimensional arrays for simplified spatial problems, two-dimensional matrices for image-based applications, or higher-dimensional tensors for complex multi-modal scenarios.

[0055] Figure 4 illustrates a flow 400 for processing heatmap data through multiple computational steps, according to an embodiment. The flow 400 begins with a step 402 where a keypoint heatmap is received. Step 402 represents the input stage where the system obtains the heatmap data that requires processing for accurate spatial localization. The keypoint heatmap received at step 402 can contain multiple peaks, noise artifacts, or ambiguous spatial information that can compromise traditional SoftArgMax operations. For example, the heatmap can exhibit dual peaks when the neural network is uncertain about left-right distinctions between joints, or it can contain spurious peaks introduced by environmental noise or imaging artifacts. The flow 400 demonstrates the systematic approach to addressing these multi-peak challenges through a series of computational transformations designed to improve localization accuracy.

[0056] In some implementations, the system identifies a peak position having a maximum intensity value in the heatmap using a first operation. In particular, the first operation returns an index of the maximum intensity value in the heatmap. The peak14184066260 1PATENT identification process can be implemented using various computational approaches including linear search algorithms that sequentially examine each position in the heatmap to find the maximum value, vectorized operations that leverage parallel processing capabilities to simultaneously compare multiple values, or other approaches. For one-dimensional heatmaps, the first operation can return a single integer index corresponding to the position of the maximum value, while for two- dimensional heatmaps, it can return coordinate pairs (x, y) indicating the pixel location with highest intensity. In higher-dimensional applications, the operation can return multi-dimensional coordinate tuples that specify the exact location within the tensor structure. The platform can handle edge cases such as tie-breaking scenarios where multiple positions share the same maximum value through deterministic rules, such as selecting the first occurrence in lexicographic order, the last occurrence, or the position closest to a predetermined reference point. Additionally, the peak identification can incorporate noise filtering or smoothing operations to ensure robust detection in the presence of spurious intensity spikes or measurement artifacts.

[0057] As shown in Figure 4, at step 404, the platform performs a first operation on the heatmap. The peak position represents the spatial coordinates where the heatmap exhibits its highest intensity value, indicating the most probable location of the target feature or joint. The index returned by the first operation can be a single integer for one-dimensional heatmaps, a coordinate pair (x, y) for two-dimensional heatmaps, or a multi-dimensional coordinate tuple for higher-dimensional applications. For example, in a 64x64 pixel heatmap representing joint probability distributions, the first operation can return coordinates such as (32, 45), indicating the pixel location with maximum intensity. The identification process can be implemented using various computational approaches including linear search algorithms for small datasets, optimized vectorized operations for medium-sized arrays, or parallel processing techniques for large-scale heatmaps.

[0058] In particular, the first operation can be an ArgMax function that identifies indices of the peak position by determining a location of the maximum intensity value within the heatmap. The ArgMax function is a mathematical operation that returns the argument (index or coordinate) at which a given function achieves its maximum value. For example, given a vector [0.2, 0.8, 0.3, 0.9, 0.1 ], the ArgMax function returns index 3 because the value 0.9 at that position is the maximum. In the context of heatmaps, ArgMax operations can be applied along specific dimensions or across the entire array15184066260 1PATENT structure. The function can be implemented using various algorithms including sequential scanning for guaranteed accuracy, heap-based approaches for efficiency with large datasets, or approximate methods using sampling techniques for real-time applications. Moreover, the ArgMax operation can handle tie-breaking scenarios where multiple positions share the same maximum value through deterministic rules such as selecting the first occurrence, the last occurrence, or the position closest to a reference point.

[0059] In some implementations, the system generates a filtered heatmap by (i) setting intensity values beyond a predetermined distance threshold from the peak position to zero and (ii) retaining intensity values within the predetermined distance threshold. The filtering technique operates by creating a spatial mask centered on the identified peak position, where the mask defines a region of interest based on the chosen distance metric and threshold value. The predetermined distance threshold can be defined using various distance metrics including Euclidean distance for circular filtering regions, Manhattan distance for diamond-shaped regions, or Chebyshev distance for square-shaped regions. For example, with a Euclidean distance threshold of 10 pixels, all intensity values located more than 10 pixels away from the peak position are set to zero, while values within this radius are preserved. The filtering process effectively isolates the primary peak region while eliminating interference from secondary peaks, noise artifacts, and spurious activations that could compromise the accuracy of subsequent SoftArgMax computations. The threshold value can be determined through empirical analysis of the specific application domain, cross- validation techniques using ground truth data, or adaptive algorithms that adjust based on heatmap characteristics such as peak sharpness, signal-to-noise ratio, or the spatial distribution of intensity values across the heatmap.

[0060] In some implementations, generating the filtered heatmap includes calculating distances from each spatial position in the heatmap to the peak position, comparing each calculated distance to the predetermined distance threshold, and selectively zeroing intensity values at spatial positions for which the distances exceed the predetermined distance threshold. The distance calculation process involves computing the spatial separation between every pixel or coordinate in the heatmap and the identified peak position using the chosen distance metric. The comparison operation evaluates whether each calculated distance exceeds the predetermined threshold, creating a binary mask that identifies which positions should be retained16184066260 1PATENT versus zeroed. The selective zeroing process applies this mask to the original heatmap, preserving intensity values where the mask indicates proximity to the peak while setting distant values to zero. This approach can be optimized using vectorized operations for computational efficiency, parallel processing for large heatmaps, or lookup tables for frequently used threshold values.

[0061] As shown in Figure 4, this corresponds to step 406, where the flow 400 keeps only values close to the ArgMax. Step 406 represents a transformation stage in the processing pipeline, where the original multi-peak heatmap received at step 402 is converted into a localized representation focused on the most prominent feature. The visual representation in Figure 4 demonstrates how the filtering operation at step 406 dramatically reduces the complexity of the heatmap by eliminating distant peaks and background noise, creating a cleaner signal for the subsequent SoftArgMax computation at step 408. Figure 4 illustrates the effect of this filtering, showing how the retained intensity values form a concentrated distribution around the peak location while distant values are completely eliminated. This processing step ensures that the weighted averaging performed by the SoftArgMax operation in step 408 focuses exclusively on the most relevant spatial region, preventing the blending effects that would occur if multiple distant peaks were included in the computation. The filtered heatmap represents a processed version of the original heatmap where distant peaks and noise artifacts have been eliminated to focus computational resources on the most relevant spatial region, thereby enabling more precise and computationally efficient localization while maintaining the differentiability required for neural network training.

[0062] In some implementations, the system applies a second operation to the filtered heatmap to compute a weighted average of spatial positions based on normalized exponential values of the intensity values within the predetermined distance threshold. In particular, the second operation transforms the filtered heatmap into a probability distribution and computes a weighted sum of indices based on the probability distribution. As shown in Figure 4, this corresponds to step 408, where the flow 400 computes a SoftArgMax operation on the filtered values. The weighted average computation provides a differentiable approximation of the peak location that enables gradient-based optimization during neural network training. The normalized exponential values are computed by applying an exponential function to each retained intensity value and then normalizing the results to sum to unity, creating a valid probability distribution. The weighted sum multiplies each spatial position by its17184066260 1PATENT corresponding probability and sums the results to produce the final coordinate estimate.

[0063] In particular, the second operation includes a SoftArgMax function that applies an exponential function to each retained intensity value, normalizes exponential values generated by the exponential function to create the probability distribution, and computes the weighted sum by multiplying each spatial position by its corresponding probability. The SoftArgMax function represents a differentiable approximation of the ArgMax operation that enables gradient computation for neural network training while providing smooth, continuous outputs. The exponential function amplifies differences between intensity values, making higher values more prominent in the final probability distribution. The normalization step divides each exponential value by the sum of all exponential values, ensuring the resulting probabilities sum to 1 .0. This approach can be implemented using various computational frameworks including TensorFlow operations for GPU acceleration, NumPy functions for CPUbased processing, or custom implementations optimized for specific hardware architectures.

[0064] Figure 5 depicts a graph 500 showing signal intensity variations with SoftArgMax operations, according to aspects of the present disclosure. The graph 500 displays two prominent peaks — one smaller peak near the beginning of the range and a larger peak near the end, with two vertical dashed lines labeled “SoftArgMax” and “Localized SoftArgMax” positioned at different points along the x-axis. The y-axis shows values ranging from 0 to approximately 0.06, while the x-axis extends from 0 to 250. This visualization demonstrates the fundamental problem addressed by the localized normalized exponential function: traditional SoftArgMax operations can produce outputs that fall between multiple peaks rather than accurately identifying the most prominent peak. The “SoftArgMax” line shows where a conventional SoftArgMax operation would place the estimated position, which falls between the two peaks due to the weighted averaging nature of the operation. In contrast, the “Localized SoftArgMax” line shows the improved accuracy achieved by the localized approach, which focuses on the dominant peak and produces a more accurate position estimate.

[0065] In some implementations, the system output, via the computing interface, coordinates corresponding to the weighted average as a localized position estimate for joint localization in the pose estimation application. In some implementations, the coordinates can be output for the spatial localization application. The localized position18184066260 1PATENT estimate represents the final result of the computational process, providing precise spatial coordinates that indicate the most probable location of the target feature or joint. The coordinates can be formatted in various representations including integer pixel coordinates for direct image processing applications, floating-point coordinates for sub-pixel precision, or normalized coordinates scaled to standard ranges for crossplatform compatibility. The computing interface used for output can include the same mechanisms used for input, such as APIs for programmatic access, file systems for persistent storage, network connections for distributed processing, or direct memory interfaces for real-time applications. The joint localization results can be integrated into larger pose estimation pipelines for applications such as motion capture, humancomputer interaction, sports analysis, or medical assessment.

[0066] Returning to Figure 3, the pose estimation diagram 300 illustrates how the localized normalized exponential function can address the challenges of multi-peak heatmaps in practical pose estimation scenarios. The right knee position 304 and right knee estimation 308 demonstrate situations where traditional SoftArgMax operations can struggle with left-right ambiguities. When the neural network exhibits uncertainty about spatial distinctions, it can produce heatmaps with multiple peaks for each joint location. The localized normalized exponential function addresses these challenges by applying the filtering and weighted averaging techniques described herein to eliminate the interference from secondary peaks and noise artifacts. By identifying the most prominent peak in the heatmap and applying the predetermined distance threshold to retain only relevant intensity values, the system can focus on the true anatomical location rather than being misled by ambiguous signals. In some cases, the adaptive threshold mechanism can further refine the filtering parameters based on feedback data, enabling the system to learn optimal localization strategies for different joint types and pose configurations. As a result of these computational improvements, the right knee estimation 308 can converge to the same spatial coordinates as the actual location of the right knee position 304, demonstrating how the localized approach can achieve precise joint localization by eliminating the blending effects that compromise traditional SoftArgMax operations in multi-peak scenarios.

[0067] In some implementations, the system receives feedback data indicating an accuracy of the localized position estimate compared to ground truth joint positions. The feedback data provides quantitative measures of localization performance that can be used to evaluate and improve the system’s accuracy over time. Ground truth19184066260 1PATENT joint positions represent the actual, verified locations of anatomical joints, typically obtained through manual annotation by experts, motion capture systems with high precision sensors, or other reliable measurement techniques. For example, ground truth data can be collected using optical motion capture systems that track reflective markers placed on specific anatomical landmarks with sub-millimeter accuracy. The accuracy metrics can include various measures such as Euclidean distance errors between predicted and actual positions, percentage of predictions within acceptable error thresholds, or normalized accuracy scores adjusted for image resolution and subject size. The feedback data can be collected in real time during system operation for immediate performance assessment or in batch mode for comprehensive evaluation across large datasets. This feedback mechanism enables continuous learning and adaptation of the localized normalized exponential function parameters.

[0068] In some implementations, the system adjusts the predetermined distance threshold based on the feedback data. This adaptive capability enables the system to optimize its filtering parameters based on observed performance, creating a selfimproving mechanism that can adapt to different pose estimation scenarios and environmental conditions. The adjustment process can involve various optimization techniques including gradient-based methods that minimize localization error, grid search approaches that evaluate discrete threshold values, or reinforcement learning algorithms that learn optimal thresholds through trial and error. For example, if the feedback data indicates that localization errors are consistently high, the system can increase the distance threshold to include more spatial context around the peak, potentially capturing relevant information that was previously filtered out. Conversely, if the system detects that noise artifacts are degrading performance, it can decrease the threshold to focus more tightly on the primary peak region. The adjustment mechanism can operate at different timescales, from real-time adaptation for dynamic environments to periodic updates based on accumulated performance data.

[0069] In some implementations, adjusting the predetermined distance threshold includes analyzing the feedback data to determine localization errors between the localized position estimate and the ground truth joint positions, based on determining that the localization errors exceed a first error threshold, increasing the predetermined distance threshold, and based on determining that the localization errors fall below a second error threshold, decreasing the predetermined distance threshold. The error analysis process involves computing statistical measures of localization accuracy,20184066260 1PATENT such as mean absolute error, root mean square error, or percentile-based error metrics, across multiple predictions. The first error threshold represents a performance level below which the system determines that the current filtering approach is too restrictive, potentially excluding relevant spatial information that could improve accuracy. For example, if the mean localization error exceeds 5 pixels, the system can increase the distance threshold from 10 pixels to 12 pixels to include more contextual information. The second error threshold represents a performance level above which the system determines that the filtering is too permissive, potentially including noise or irrelevant peaks that degrade accuracy. The threshold adjustment mechanism can implement hysteresis to prevent oscillatory behavior, ensuring that the system maintains stable performance while adapting to changing conditions. Moreover, the adjustment process can incorporate domain-specific knowledge about typical joint movement patterns, anatomical constraints, and application requirements to guide the optimization process toward practically useful solutions.

[0070] Returning to Figure 3, as the localized normalized exponential function continues to refine its accuracy through adaptive threshold adjustments, the alignment between estimated joint positions and actual joint positions in pose estimation diagram 300 can improve significantly. The left knee estimation 306 and right knee estimation 308 can converge more closely to their corresponding actual positions as the system learns optimal filtering parameters for different anatomical landmarks. In some cases, the adaptive mechanism can enable the system to distinguish more effectively between similar joint locations, reducing confusion between left and right joint pairs that can occur in traditional SoftArgMax approaches. The pose estimation diagram 300 demonstrates how the localized approach can maintain spatial consistency across multiple joints within the same pose estimation framework. As the system accumulates feedback data and adjusts its parameters accordingly, the overall skeletal structure representation can become more anatomically accurate, with joint estimations that better reflect the true biomechanical constraints and movement patterns of human pose dynamics.Methodologies for Pose Estimation

[0071] Figure 6 includes a flow diagram of a process 600 for pose estimation. Initially, the pose estimation platform can receive, via a computing interface, a heatmap indicating intensity values at spatial positions for a spatial localization application (step 601 ). For example, the platform can obtain heatmap data generated21184066260 1PATENT by a neural network that represents probability distributions of joint locations within an image frame. The process can then identify a peak position having a maximum intensity value in the heatmap using a first operation (step 602). In some embodiments, this first operation can include an ArgMax function that determines the location of the maximum intensity value, providing the most probable spatial coordinates for the target feature or joint.

[0072] Then, the process can generate a filtered heatmap by setting intensity values beyond a predetermined distance threshold from the peak position to a reduced value while retaining intensity values within the predetermined distance threshold (step 603). For example, the system can calculate distances from each spatial position to the identified peak and selectively zero out values that exceed the threshold, effectively eliminating distant peaks and noise artifacts. The process can apply a second operation to the filtered heatmap to compute a weighted average of spatial positions based on normalized exponential values of the intensity values within the predetermined distance threshold (step 604). In some cases, this second operation can include a SoftArgMax function that transforms the filtered heatmap into a probability distribution and computes a weighted sum of indices to provide a differentiable approximation of the peak location.

[0073] Finally, the process can output, via the computing interface, coordinates corresponding to the weighted average as a localized position estimate for the spatial localization application (step 605). For example, the system can provide precise spatial coordinates that indicate the most probable location of the target feature or joint, formatted for integration into larger pose estimation pipelines for applications such as motion capture, human-computer interaction, or medical assessment. In some embodiments, these coordinates can be expressed in various representations including pixel coordinates for direct image processing or normalized coordinates for cross-platform compatibility.Example Machine Learning Model

[0074] Figure 7 illustrates an example machine learning model 702. The machine learning model 702 can be used for analyzing pose data, identifying joint positions, or generating pose estimation results in the pose estimation platform. According to some examples, the machine learning model can be any model. In some embodiments, the machine learning model 702 can be trained to intake input 704, including input data received. As a result of inputting the input 704 into the machine learning model 702,22184066260 1PATENT the machine learning model 702 can then output an output 706. As described herein, the input data can include data such as image frames, video sequences, sensor data from motion capture devices, depth information, or skeletal structure data. In some embodiments, the output 706 can include detected joint positions, pose classifications, skeletal structure representations, confidence scores, or movement tracking results.

[0075] The output parameters can be fed back to the machine learning model 702 as input to train the machine learning model 702 (e.g., alone or in conjunction with user indications of the accuracy of outputs, labels associated with the inputs, or other reference feedback information). The machine learning model 702 can update its configurations (e.g., weights, biases, or other parameters) based on the assessment of its prediction and reference feedback information (e.g., user indication of accuracy, reference labels, or other information). Connection weights can be adjusted, for example, if the machine learning model 702 is a neural network, to reconcile differences between the neural network’s prediction and the reference feedback.

[0076] One or more neurons of a neural network can require that their respective errors be sent backward through the neural network to facilitate the update process (e.g., backpropagation of error). Updates to the connection weights may, for example, be reflective of the magnitude of error propagated backward after a forward pass has been completed. In this way, for example, the machine learning model can be trained to generate better predictions for pose detection and joint localization.

[0077] In some embodiments, the machine learning model 702 can include an artificial neural network. In such embodiments, the machine learning model 702 can include an input layer and one or more hidden layers. Each neural unit of the machine learning model 702 can be connected to one or more other neural units of the machine learning model 702. Such connections can be enforcing or inhibitory in their effect on the activation state of connected neural units. Each individual neural unit can have a summation function that combines the values of all of its inputs together. Each connection (or the neural unit itself) can have a threshold function that a signal must surpass before it propagates to other neural units. The machine learning model 702 can be self-learning or trained rather than explicitly programmed and can perform significantly better in certain areas of problem solving as compared to computer programs that do not use machine learning. During training, an output layer of the machine learning model 702 can correspond to a classification of poses or joint positions, and an input known to correspond to that classification can be input into an23184066260 1PATENT input layer of the machine learning model 702 during training. During testing, an input without a known classification can be input into the input layer, and a determined classification can be output.

[0078] The machine learning model 702 can include embedding layers in which each feature of a vector is converted into a dense vector representation. These dense vector representations for each feature can be pooled at one or more subsequent layers to convert the set of embedding vectors into a single vector. The machine learning model 702 can be structured as a factorization machine model. The machine learning model 702 can be a nonlinear model or supervised learning model that can perform classification or regression for pose detection and joint localization. For example, the machine learning model 702 can be a general-purpose supervised learning algorithm that the pose estimation platform uses for both classification and regression tasks related to pose estimation and movement tracking. Alternatively, the machine learning model 702 can include a Bayesian model configured to perform variational inference on the pose data or joint position vectors.

[0079] To assist in understanding the present disclosure, some concepts relevant to neural networks and machine learning are discussed herein. Generally, a neural network includes a number of computation units (sometimes referred to as “neurons”). Each neuron receives an input value and applies a function to the input to generate an output value. The function typically includes a parameter (also referred to as a “weight”) whose value is learned through the process of training. A plurality of neurons can be organized into a neural network layer (or simply “layer”), and there can be multiple such layers in a neural network. The output of one layer can be provided as input to a subsequent layer. Thus, input to a neural network can be processed through a succession of layers until an output of the neural network is generated by a final layer. This is a simplistic discussion of neural networks, and there can be more complex neural network designs that include feedback connections, skip connections, or other such possible connections between neurons or layers, which are not discussed in detail here.

[0080] A deep neural network (DNN) is a type of neural network having multiple layers or a large number of neurons. The term DNN can encompass any neural network having multiple layers, including convolutional neural networks (CNNs), recurrent neural networks (RNNs), multilayer perceptrons (MLPs), Generative24184066260 1PATENTAdversarial Networks (GANs), Variational Autoencoders (VAEs), and auto-regressive models, among others.

[0081] DNNs are often used as machine learning-based models for modeling complex behaviors (e.g., pose data analysis, joint position detection, movement tracking, etc.) in order to improve the accuracy of outputs (e.g., more accurate pose classifications, more precise joint localizations), for example, as compared with models with fewer layers. In the present disclosure, the term “machine learning-based model” or, more simply, “machine learning model” can refer to a DNN. Training a machine learning model refers to a process of learning the values of the parameters (or weights) of the neurons in the layers such that the machine learning model is able to model the target behavior to a desired degree of accuracy. Training typically requires the use of a training dataset, which is a set of data that is relevant to the target behavior of the machine learning model.

[0082] Training a machine learning model generally involves inputting into a machine learning model (e.g., an untrained machine learning model) training data to be processed by the machine learning model, processing the training data using the machine learning model, collecting the output generated by the machine learning model (e.g., based on the inputted training data), and comparing the output to a desired set of target values. If the training data is labeled, the desired target values can be, e.g., the ground truth labels of the training data, such as validated pose classifications or expert-annotated joint positions. If the training data is unlabeled, the desired target value can be a reconstructed (or otherwise processed) version of the corresponding machine learning model input (e.g., in the case of an autoencoder) or can be a measure of some target observable effect on the environment such as pose detection accuracy metrics or joint localization precision improvements (e.g., in the case of a reinforcement learning agent). The parameters of the machine learning model are updated based on a difference between the generated output value and the desired target value. For example, if the value outputted by the machine learning model is excessively high, the parameters can be adjusted so as to lower the output value in future training iterations. An objective function is a way to quantitatively represent how close the output value is to the target value. An objective function represents a quantity (or one or more quantities) to be optimized (e.g., minimize a loss or maximize a reward) in order to bring the output value as close to the target value25184066260 1PATENT as possible. The goal of training the machine learning model is typically to minimize a loss function, or a reconstruction loss, or maximize a reward function.

[0083] The training data can be a subset of a larger dataset. For example, a dataset can be split into three mutually exclusive subsets: a training set, a validation (or cross-validation) set, and a testing set. The three subsets of data can be used sequentially during machine learning model training. For example, the training set can be first used to train one or more machine learning models, each machine learning model, e.g., having a particular architecture, having a particular training procedure, being describable by a set of model hyperparameters, or otherwise being varied from the other of the one or more machine learning models. The validation (or cross- validation) set can then be used as input data into the trained machine learning models to, e.g., measure the performance of the trained machine learning models or compare performance between them. Where hyperparameters are used, a new set of hyperparameters can be determined based on the measured performance of one or more of the trained machine learning models, and the first step of training (e.g., with the training set) can begin again on a different machine learning model described by the new set of determined hyperparameters. In this way, these steps can be repeated to produce a more performant trained machine learning model. Once such a trained machine learning model is obtained (e.g., after the hyperparameters have been adjusted to achieve a desired level of performance), a third step of collecting the output generated by the trained machine learning model applied to the third subset (the testing set) can begin. The output generated from the testing set can be compared with the corresponding desired target values to give a final assessment of the trained machine learning model’s accuracy. Other segmentations of the larger dataset or schemes for using the segments for training one or more machine learning models are possible.

[0084] Backpropagation is an algorithm for training a machine learning model. Backpropagation is used to adjust (e.g., update) the value of the parameters in the machine learning model with the goal of optimizing the objective function. For example, a defined loss function is calculated by forward propagation of an input to obtain an output of the machine learning model and a comparison of the output value with the target value. Backpropagation calculates a gradient of the loss function with respect to the parameters of the machine learning model, and a gradient algorithm (e.g., gradient descent) is used to update (e.g., “learn”) the parameters to reduce the26184066260 1PATENT loss function. Backpropagation is performed iteratively so that the loss function is converged or minimized. Other techniques for learning the parameters of the machine learning model can be used. The process of updating (or learning) the parameters over many iterations is referred to as training. Training can be carried out iteratively until a convergence condition is met (e.g., a predefined maximum number of iterations has been performed, or the value outputted by the machine learning model is sufficiently converged with the desired target value), after which the machine learning model is considered to be sufficiently trained. The values of the learned parameters can then be fixed, and the machine learning model can be deployed to generate output in real-world applications (also referred to as “inference”).

[0085] In some examples, a trained machine learning model can be fine-tuned, meaning that the values of the learned parameters can be adjusted slightly in order for the machine learning model to better model a specific task. Fine-tuning of a machine learning model typically involves further training the machine learning model on a number of data samples (which can be smaller in number / cardinality than those used to train the model initially) that closely target the specific task, such as domainspecific pose estimation tasks or particular movement patterns in specialized pose detection applications.Processing System

[0086] Figure 8 is a block diagram illustrating an example of a processing system 800 that can perform at least some operations described herein. For example, some components of the processing system 800can be hosted on a computing device that includes a pose estimation platform (e.g., pose estimation platform 102 of Figure 1 or pose estimation platform 212 of Figure 2).

[0087] The processing system 800 can include a processor 802, a main memory 806, a non-volatile memory 810, a network adapter 812, a display mechanism 818, an input / output device 820, a control device 822 (e.g., a keyboard or pointing device), a drive unit 824 including a storage medium 826, and a signal generation device 830 that are communicatively connected to a bus 816. The bus 816 is illustrated as an abstraction that represents one or more physical buses or point-to-point connections that are connected by appropriate bridges, adapters, or controllers. The bus 816, therefore, can include a system bus, a Peripheral Component Interconnect (PCI) bus or a PCI-Express bus, a HyperTransport or an industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, a universal serial bus (USB), an27184066260 1PATENT inter-integrated circuit (l2C) bus, or an IEEE standard 1384 bus (also referred to as “Firewire”).

[0088] While the main memory 806, non-volatile memory 810, and storage medium 826 are shown to be a single medium, the terms “machine-readable medium” and “storage medium” should be taken to include a single medium or multiple media (e.g., a centralized / distributed database and / or associated caches and servers) that store one or more sets of instructions 828. The terms “machine-readable medium” and “storage medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the processing system 800.

[0089] In general, the routines executed to implement the embodiments of the disclosure can be implemented as part of an operating system or a specific application, component, program, object, module, or sequence of instructions (collectively referred to as “computer programs”). The computer programs typically include one or more instructions (e.g., instructions 804, 808, 828) set at various times in various memory and storage devices in a computing device. When read and executed by the processor 802, the instruction(s) cause the processing system 800 to perform operations to execute elements involving the various aspects of the present disclosure.

[0090] Further examples of machine- and computer-readable media include recordable-type media, such as volatile memory devices and non-volatile memory 810, removable disks, hard disk drives, and optical disks (e.g., Compact Disc Read- Only Memory (CD-ROMs) and Digital Versatile Discs (DVDs)), and transmission-type media, such as digital and analog communication links.

[0091] The network adapter 812 enables the processing system 800 to mediate data in a network 814 with an entity that is external to the processing system 800 through any communication protocol supported by the processing system 800 and the external entity. The network adapter 812 can include a network adapter card, a wireless network interface card, a router, an access point, a wireless router, a switch, a multilayer switch, a protocol converter, a gateway, a bridge, a bridge router, a hub, a digital media receiver, a repeater, or any combination thereof.Remarks

[0092] The foregoing description of various embodiments of the claimed subject matter has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the claimed subject matter to the precise forms disclosed. Many modifications and variations will be apparent to one skilled in the art.28184066260 1PATENTEmbodiments were chosen and described in order to best describe the principles of the invention and its practical applications, thereby enabling those skilled in the relevant art to understand the claimed subject matter, the various embodiments, and the various modifications that are suited to the particular uses contemplated.

[0093] Although the Detailed Description describes certain embodiments and the best mode contemplated, the technology can be practiced in many ways, no matter how detailed the Detailed Description appears. Embodiments can vary considerably in their implementation details while still being encompassed by the specification. Particular terminology used when describing certain features or aspects of various embodiments should not be taken to imply that the terminology is being redefined herein to be restricted to any specific characteristics, features, or aspects of the technology with which that terminology is associated. In general, the terms used in the following claims should not be construed to limit the technology to the specific embodiments disclosed in the specification unless those terms are explicitly defined herein. Accordingly, the actual scope of the technology encompasses not only the disclosed embodiments but also all equivalent ways of practicing or implementing the embodiments.

[0094] The language used in the specification has been principally selected for readability and instructional purposes. It may not have been selected to delineate or circumscribe the subject matter. It is therefore intended that the scope of the technology be limited not by this Detailed Description but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of various embodiments is intended to be illustrative, but not limiting, of the scope of the technology as set forth in the following claims.29184066260 1

Claims

PATENTCLAIMSWhat Is Claimed Is:1 . A computer-implemented method for improving spatial localization accuracy in neural networks, the computer-implemented method comprising: receiving, via a computing interface, a heatmap indicating intensity values at spatial positions, wherein the heatmap is generated by a neural network for a pose estimation application; identifying a peak position having a maximum intensity value in the heatmap using a first operation, wherein the first operation returns an index of the maximum intensity value in the heatmap; generating a filtered heatmap by (i) setting intensity values beyond a predetermined distance threshold from the peak position to zero and (ii) retaining intensity values within the predetermined distance threshold; applying a second operation to the filtered heatmap to compute a weighted average of spatial positions based on normalized exponential values of the intensity values within the predetermined distance threshold, wherein the second operation transforms the filtered heatmap into a probability distribution and computes a weighted sum of indices based on the probability distribution; outputting, via the computing interface, coordinates corresponding to the weighted average as a localized position estimate for joint localization in the pose estimation application; receiving feedback data indicating an accuracy of the localized position estimate compared to ground truth joint positions; and adjusting the predetermined distance threshold based on the feedback data.

2. The computer-implemented method of claim 1 , wherein the first operation includes an ArgMax function that identifies indices of the peak position by determining a location of the maximum intensity value within the heatmap.30184066260 1PATENT3. The computer-implemented method of claim 1 , wherein the second operation includes a SoftArgMax function that applies an exponential function to each retained intensity value, normalizes exponential values generated by the exponential function to create the probability distribution, and computes the weighted sum by multiplying each spatial position by its corresponding probability.

4. The computer-implemented method of claim 1 , wherein generating the filtered heatmap includes: calculating distances from each spatial position in the heatmap to the peak position; comparing each calculated distance to the predetermined distance threshold; and selectively zeroing intensity values at spatial positions for which the distances exceed the predetermined distance threshold.

5. The computer-implemented method of claim 1 , wherein the heatmap includes a two-dimensional array representing joint probability distributions for human pose estimation, and wherein the localized position estimate corresponds to predicted coordinates of a specific anatomical joint within an image frame.

6. The computer-implemented method of claim 1 , wherein adjusting the predetermined distance threshold includes: analyzing the feedback data to determine localization errors between the localized position estimate and the ground truth joint positions; based on determining that the localization errors exceed a first error threshold, increasing the predetermined distance threshold; and based on determining that the localization errors fall below a second error threshold, decreasing the predetermined distance threshold.

7. A computer-implemented method comprising: receiving, via a computing interface, a heatmap indicating intensity values at spatial positions, wherein the heatmap is generated by a neural network for a spatial localization application;31184066260 1PATENT identifying a peak position having a maximum intensity value in the heatmap using a first operation, wherein the first operation returns an index of the maximum intensity value in the heatmap; generating a filtered heatmap by (i) setting intensity values beyond a predetermined distance threshold from the peak position to a reduced value and (ii) retaining intensity values within the predetermined distance threshold; applying a second operation to the filtered heatmap to compute a weighted average of spatial positions based on normalized exponential values of the intensity values within the predetermined distance threshold, wherein the second operation transforms the filtered heatmap into a probability distribution and computes a weighted sum of indices based on the probability distribution; and outputting, via the computing interface, coordinates corresponding to the weighted average as a localized position estimate for the spatial localization application.

8. The computer-implemented method of claim 7, wherein the first operation includes an ArgMax function that identifies indices of the peak position by determining a location of the maximum intensity value within the heatmap.

9. The computer-implemented method of claim 7, wherein the second operation includes a SoftArgMax function that applies an exponential function to each retained intensity value, normalizes exponential values generated by the exponential function to create the probability distribution, and computes the weighted sum by multiplying each spatial position by its corresponding probability.

10. The computer-implemented method of claim 7, wherein generating the filtered heatmap includes: calculating distances from each spatial position in the heatmap to the peak position; comparing each calculated distance to the predetermined distance threshold; and32184066260 1PATENT selectively zeroing intensity values at spatial positions for which the distances exceed the predetermined distance threshold.11 . The computer-implemented method of claim 7, wherein the heatmap includes a two-dimensional array representing joint probability distributions for human pose estimation, and wherein the localized position estimate corresponds to predicted coordinates of a specific anatomical joint within an image frame.

12. The computer-implemented method of claim 7, further comprising: receiving feedback data indicating an accuracy of the localized position estimate compared to ground truth joint positions; and adjusting the predetermined distance threshold based on the feedback data.

13. The computer-implemented method of claim 12, wherein adjusting the predetermined distance threshold includes: analyzing the feedback data to determine localization errors between the localized position estimate and the ground truth joint positions; based on determining that the localization errors exceed a first error threshold, increasing the predetermined distance threshold; and based on determining that the localization errors fall below a second error threshold, decreasing the predetermined distance threshold.

14. One or more non-transitory media with instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform operations comprising: receiving, via a computing interface, a heatmap indicating intensity values at spatial positions for a spatial localization application; identifying a peak position having a maximum intensity value in the heatmap using a first operation; generating a filtered heatmap by (i) setting intensity values beyond a predetermined distance threshold from the peak position to a reduced value and (ii) retaining intensity values within the predetermined distance threshold;33184066260 1PATENT applying a second operation to the filtered heatmap to compute a weighted average of spatial positions based on normalized exponential values of the intensity values within the predetermined distance threshold; and outputting, via the computing interface, coordinates corresponding to the weighted average as a localized position estimate for the spatial localization application.

15. The one or more non-transitory media of claim 14, wherein the first operation includes an ArgMax function that identifies indices of the peak position by determining a location of the maximum intensity value within the heatmap.

16. The one or more non-transitory media of claim 14, wherein the second operation includes a SoftArgMax function that applies an exponential function to each retained intensity value, normalizes exponential values generated by the exponential function to create a probability distribution, and computes a weighted sum by multiplying each spatial position by its corresponding probability.

17. The one or more non-transitory media of claim 14, wherein generating the filtered heatmap includes: calculating distances from each spatial position in the heatmap to the peak position; comparing each calculated distance to the predetermined distance threshold; and selectively zeroing intensity values at spatial positions for which the distances exceed the predetermined distance threshold.

18. The one or more non-transitory media of claim 14, wherein the heatmap includes a two-dimensional array representing joint probability distributions for human pose estimation, and wherein the localized position estimate corresponds to predicted coordinates of a specific anatomical joint within an image frame.

19. The one or more non-transitory media of claim 14, further comprising : receiving feedback data indicating an accuracy of the localized position estimate compared to ground truth joint positions; and34184066260 1PATENT adjusting the predetermined distance threshold based on the feedback data.

20. The one or more non-transitory media of claim 19, wherein adjusting the predetermined distance threshold includes: analyzing the feedback data to determine localization errors between the localized position estimate and the ground truth joint positions; based on determining that the localization errors exceed a first error threshold, increasing the predetermined distance threshold; and based on determining that the localization errors fall below a second error threshold, decreasing the predetermined distance threshold.35184066260 1

Images