Spatio-temporal 3-d mesh smoothing

The spatiotemporal mesh smoothing method addresses shape distortion and computation challenges by integrating a remesher and two-model smoother to maintain mesh topology and reduce vertex jittering, enhancing accuracy and flexibility in 3-D mesh applications.

WO2026239167A1PCT designated stage Publication Date: 2026-11-19SONY GROUP CORP +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/IB2026/050324
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-12
Filing Date
2026-01-14
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Traditional temporal smoothing methods for 3-D meshes often lead to unexpected shape changes due to focusing on individual vertex positions without considering mesh topology, and motion learning involves heavy computation and data collection challenges.

Method used

A spatiotemporal mesh smoothing method that combines a remesher with a two-model smoother, incorporating a pure spatial and spatio-temporal model to maintain mesh topology consistency and reduce vertex jittering.

Benefits of technology

The method effectively reduces vertex jittering and maintains mesh accuracy without distorting shape properties, offering a lightweight and flexible solution applicable to various 3-D mesh applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure IB2026050324_19112026_PF_FP_ABST
    Figure IB2026050324_19112026_PF_FP_ABST
Patent Text Reader

Abstract

Temporal mesh smoothing, including: receiving an initial mesh and a sequence of raw meshes; calculating an estimated mesh at a vertex and a particular time as most likely mesh that best explains the sequence of raw meshes and neighbors of the vertex given estimated meshes calculated at previous in time to the particular time; calculating a pure spatial part by calculating a conditional probability of the estimated mesh given a position of the vertex at the particular time; and calculating a spatio-temporal part by estimating the position of the vertex considering positions of previous neighboring vertices in time.
Need to check novelty before this filing date? Find Prior Art

Description

SPATIO-TEMPORAL 3-D MESH SMOOTHINGBACKGROUNDField

[0001] The present disclosure relates to temporal mesh smoothing, and more specifically to spatio-temporal 3-D mesh smoothing.Background

[0002] Temporal smoothing on 3 -D discrete point data may be used to reduce noise and highlight underlying trends in the sequence of 3 -D point cloud data over time. This can improve the accuracy of 3 -D object tracking, pose estimation, and other applications. Several techniques, including filtering, interpolation, and machine learning methods, can be employed for this purpose.

[0003] However, temporal smoothing using traditional filtering which focuses on smoothing the individual vertex positions without considering the mesh topology may lead to unexpected shape changes. Further, the motion learning may involve heavy computation and model size, which may be difficult to collect the training data fully covering all local shape dynamics.SUMMARY

[0004] The present disclosure provides for spatiotemporal mesh smoothing.

[0005] In one implementation, a method for temporal mesh smoothing is disclosed. The method includes: receiving an initial mesh and a sequence of raw meshes; calculating an estimated mesh at a vertex and a particular time as mostAttorney Docket No. 113748-0313WO01likely mesh that best explains the sequence of raw meshes and neighbors of the vertex given estimated meshes calculated at previous in time to the particular time; calculating a pure spatial part by calculating a conditional probability of the estimated mesh given a position of the vertex at the particular time; and calculating a spatio-temporal part by estimating the position of the vertex considering positions of previous neighboring vertices in time.

[0006] In another implementation, a method for temporal mesh smoothing is disclosed. The method includes: receiving an initial mesh and a sequence of raw meshes; calculating an estimated mesh at a vertex and a particular time using the initial mesh and the sequence of raw meshes; calculating a pure spatial part by calculating a conditional probability of the estimated mesh given a position of the vertex at the particular time; calculating a spatio-temporal part; and executing an iterative form of a two-model smoother by maximizing the pure spatial part and the spatio-temporal part.

[0007] Other features and advantages should be apparent from the present description which illustrates, by way of example, aspects of the disclosure.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The details of the present disclosure, both as to its structure and operation, may be gleaned in part by study of the appended drawings, in which like reference numerals refer to like parts, and in which:

[0009] FIG. 1 is a diagram illustrating Equation [6] in accordance with one implementation of the presentAttorney Docket No. 113748-0313WO01disclosure;

[0010] FIG. 2 is a flow diagram illustrating a temporal mesh smoothing method in accordance with one implementation of the present disclosure;

[0011] FIG. 3A is an illustration of an example sequence of raw meshes or point clouds which was input to a remesher in accordance with one implementation of the present disclosure;

[0012] FIG. 3B is an illustration of an example initial mesh (MQ ) 310 which was input to the remesher in accordance with one implementation of the present disclosure; and

[0013] FIG. 3C is an illustration of an example process of the remesher in accordance with one implementation of the present disclosure.DETAILED DESCRIPTION

[0014] As described above, temporal smoothing using traditional filtering which focuses on smoothing the individual vertex positions without considering the mesh topology may lead to unexpected shape changes.Accordingly, one of the objectives of the present disclosure is to smooth a sequence of meshes continuous in time, which are in the same topology, to reduce the temporal jittering of vertices for better visual quality.

[0015] Certain implementations of the present disclosure include: 3-D remeshing of each single frame, which applies any remeshing methods, to wrap the sequence of raw meshes in time into an identical mesh topology (i. e., the same number of vertices and the same surface definitions); and using a two-model temporal smoother toAttorney Docket No. 113748-0313WO01combine a pure spatial model with a spatio-temporal model to perform smoothing on each vertex in each mesh. After reading below descriptions, it will become apparent how to implement the disclosure in various implementations and applications. Although various implementations of the present disclosure will be described herein, it is understood that these implementations are presented by way of example only, and not limitation. As such, the detailed description of various implementations should not be construed to limit the scope or breadth of the present disclosure.

[0016] In contrast to other temporal smoothing methods working on temporal positional sequence of each point, implementations of the present disclosure assign the temporal smoothing problem as a local shape spatiotemporal stabilization problem. That is, implementations of the present disclosure not only smooth trajectory of each vertex in time but keep its neighborhood on the mesh highly consistent with the shape before smoothing. More specifically, a temporal mesh smoothing method, which combines a remesher, a spatial term, and a spatiotemporal term, is proposed. In particular, the spatial term keeps each smoothed vertex close to the position calculated by remeshing, and the spatio-temporal term stimulates each smoothed vertex having the similar positional dynamics as its neighboring vertices on the predefined mesh topology.

[0017] In one implementation, solving the temporal mesh smoothing includes solving the following equation:

[0018] vi*t= maxp(iyvl,t

[0019] wherein,Attorney Docket No. 113748-0313WO01

[0020] vi,t(3x1) = unknown smoothed 3D position of vertex i at time t,

[0021] Ot= the observation data at time t for remeshing, e.g.: a raw mesh or a point cloud,

[0022] Mt= estimated mesh at time t by the remeshing pipeline,

[0023] ṽi,t∈ M̃t= estimated 3D position of vertex i at time t by the remeshing pipeline,

[0024] Mt*-= one to multiple smoothed meshes before time t,

[0025] ∈ Mt-1*= smoothed 3D position of vertex i at time t - 1,[0026= connected 3D vertex neighbors of vertex i on the mesh M,

[0027] n = |N(vi)| = number of the connected 3D vertex neighbors of vertex i,

[0029] p(M̃i,t|Ot, Mt*-, N(vi)) ~ constant, given the deterministic remeshing result M̃t.

[0030] In one implementation, solving the temporal mesh smoothing also includes separating the pure spatial part and the spatio-temporal part as follows:

[0035] is the pure spatial part, andAttorney Docket No. 113748-0313WO01

[0037] is the spatio-temporal part.

[0038] Regarding the pure spatial part, ps(M̃t|vi,t) is inversely proportional to the distance between vi,tand M̃t. Thus, the goal is to keep vi,tas close to M̃tas possible which increases parameter.

[0039] Regarding the spatio-temporal part, termsuggests the estimation of vi,tconsidering the previous (temporal) neighboring vertices' positions on the meshes (spatial). Specifically, the neighborhood shape centered at vi,ton Mt*should change smoothly along the mesh series Mt*-to suppress the singular fluctuation of vi,tin time. In the conditional term, M̃tmay be used as a surrogate of Otdue to the joint Ot, Mt*-, and N(vi), because M̃tapproximates Otin the same mesh topology on which N(vi) and Mt*-are defined to make. Accordingly, the two-model smoother output may be calculated as maximization of the pure spatial part ps(Mt|v£ t) and the spatio-temporal part[5].

[0040] In one implementation, the maximization step of Equation [5] is an iterative or single step analytical solution, depending on the design of the two models.

[0041] Example implementations for solving the temporal mesh smoothing of Equation [5] include solving the following steps:Attorney Docket No. 113748-0313WO01

[0050] wherein pt≡ p(Vi,tTVi,t).

[0051] Combining psand ptfrom the above steps:

[0059] FIG. 1 is a diagram 100 illustrating Equation [6] in accordance with one implementation of the present disclosure. The illustrated implementation of FIG. 1 shows an estimated position of vertex at time t (vi,t) 110, a smoothed position of vertex at time t (vi,t*) 120, a smoothed position of vertex at time t-1130, and a smoothed position ofAttorney Docket No. 113748-0313WO01vertex at time t-1 plus an average flow 140.

[0060] In the illustrated implementation of FIG. 1, the estimated 3D position of vertex i at time t by a meshing pipelineis represented by vertex (") and four neighbors (▲), while the smoothed 3D position of vertex i at time t — 1 by the meshing pipelineis represented by vertex ( ^ ) and four neighbors ( ■ ). Further, term N(vi) in Equation [6] may be expanded to k orders of neighbors on the mesh in the iterative form as follows:

[0063] FIG. 2 is a flow diagram illustrating a temporal mesh smoothing method 200 in accordance with one implementation of the present disclosure. In the illustrated implementation of FIG. 2, an initial mesh and a sequence of raw meshes or point clouds are received, at step 210.

[0064] Remeshing is performed, at step 220, to generate the estimated mesh at time t (Mt). In one implementation, the estimated mesh at time t may be calculated, at step 220, as:

[0065] Mt= maxp(

[0066] Wherein

[0067] Ot= observation data at time t,

[0068] Mt= maximum a posteriori estimation of the mesh at time t, given Ot,

[0069] Mt*- = past smoothed meshes before time t,Attorney Docket No. 113748-0313WO01

[0070] N(vi) = connected vertex neighbors of vertex i.

[0071] Thus, in Equation [8], the estimated mesh (Mt) is calculated by the remesher as the most likely mesh that best explains the observation data (e.g., raw meshes or point clouds) and the vertex neighbors (N(vi)), given the history of the smoothed meshes (Mt*-). In one implementation of Equation [8],of termis used as an initial mesh or as a constraint in the remesher.

[0072] FIG. 3A is an illustration of an example sequence of raw meshes or point clouds 300 which was input to a remesher (at step 210) in accordance with one implementation of the present disclosure.

[0073] FIG. 3B is an illustration of an example initial mesh (MQ ) 310 which was input to the remesher (at step 210) in accordance with one implementation of the present disclosure.

[0074] FIG. 3C is an illustration of an example process of the remesher (at step 220) in accordance with one implementation of the present disclosure. FIG. 3C shows that the remesher receives the initial mesh 310 and the sequence of raw meshes or point clouds 300 to generate a sequence of estimated meshes (Mt) 320. In the illustrated implementation of FIG. 3C, the initial mesh 310 is used as a template mesh to fit to the sequence of raw meshes 300 (which are random and inconsistent) to generate the sequence of estimated meshes 320.

[0075] Referring back to the illustrated implementation of FIG. 2, the estimated mesh (Mt) calculated by the remesher is then used to calculate, at step 230, the pure spatial part ps(M̃t|vi,t), as shown in Equation [3]. In oneAttorney Docket No. 113748-0313WO01implementation, the pure spatial part represents the conditional probability of the estimated mesh (Mt) given unknown position of vertex i at t (vi,t). That is, the pure spatial part represents the probability of M̃toccurring knowing that vi,thas already happened. In general,is inversely proportional to the distance between vi tand Mt. Thus, the goal in step 230 is to keep vi,tas close to M̃tas possible, such thatis still high. In one implementation, the two-model smoother output vi,t*=calculated asmaximization of the pure spatial part ps(Mt|v£ t) and the spatio-temporal part

[0076] In one implementation, the two-model smoother output shown above as maximization of the pure spatial part and the spatio-temporal part may be implemented in an iterative form to k orders of neighbors on the mesh as shown in Equation [7]. Thus, at step 240, the index of vertices (v) and the order of neighborhoods (k) are initialized (i.e., v0i,t= vi,t, 1 ≤ i ≤ V and k = 1, i = 1). Then, the iterative form of the two-model smoother implementation as shown (Equation 7)

[0078] is executed, at step 242.

[0079] In one implementation, the index of vertices is checked, at step 250, to determine whether the two-model smoother implementation has been executed for all vertices. If so, the order of neighborhoods is checked, atAttorney Docket No. 113748-0313WO01step 260, to determine whether the two-model smoother implementation has been executed for all neighborhoods. Otherwise, if the two-model smoother implementation has not been executed for all vertices (as determined in step 250), then the index of vertices is incremented, at step 252, and the two-model smoother implementation is executed again, at step 242. If it is determined, at step 260, that the two-model smoother implementation has been executed for all neighborhoods, then the smoothed mesh at time t (Mf ) is output, at step 270. At step 272, the time is incremented (i.e., t = t+1), and the method 200 is repeated with the remeshing step at 220. Otherwise, if the two-model smoother implementation has not been executed for all neighborhoods (as determined in step 260), then the order of neighborhoods is incremented, and the index of vertices is set to 1, at step 262, and the two-model smoother implementation is executed again, at step 242.

[0080] In combining psand ptto generate Equation [6], rT2.term oc was set to a = — - - In another implementation, as2+aT2 cthe time- and vertex-variant (Tss and CTTS are introduced into so that Equation [6] is alternativelyexpressed as:

[0082] In this implementation, the sliding temporal window technique is used to calculateto continuously update (Js^tand (fT,i,t according to the statistics of the following variables in the current temporal window,and Vi.TVi., whereinAttorney Docket No. 113748-0313WO01In one implementation, some preset &s,i,tsis used at different mesh regions according to the learned shape dynamics of the subject.

[0084] The advantages of the present implementations include:

[0085] (a) the proposed temporal smoothing method explicitly considers the underlying mesh topology, which may reduce vertex jittering (1) without compromising the mesh accuracy, (2) without distorting much shape properties, which could be subject- and location-specific and constructed by the processing steps before the smoother, and (3 ) possibly reduce the noises and achieve higher mesh accuracy than remeshing alone;

[0086] (b) the method is simple and light weight compared to other learning-based methods, and is directly applicable to online mesh streaming without any offline computations;

[0087] (c) the two models are independent from the remeshing method and does not need to be trained together, and can be tuned independently; and

[0088] (d) the choices of the spatio-temporal model and the pure spatial model are highly flexible and not limited to any of the above-disclosed implementations.

[0089] Accordingly, the above-disclosed implementations may be applied to any 4-D mesh data including: (a) digital characters / Metaverse such as the talking head animation for TV programs or the virtual front desk, augmented reality (AR) / virtual reality (VR) / extended reality (XR) video generation for commercials, games, or movies; (b) mesh compression such as a mesh smoother, which reduces the temporal high frequency noises and isAttorney Docket No. 113748-0313WO01beneficial to any mesh compressions to achieve a smaller code size; and (c) mesh streaming to be applied before or after the transmission for stable visualization.

[0090] In a particular implementation, a method for temporal mesh smoothing is disclosed. The method includes: receiving an initial mesh and a sequence of raw meshes; calculating an estimated mesh at a vertex and a particular time as most likely mesh that best explains the sequence of raw meshes and neighbors of the vertex given estimated meshes calculated at previous in time to the particular time; calculating a pure spatial part by calculating a conditional probability of the estimated mesh given a position of the vertex at the particular time; and calculating a spatio-temporal part by estimating the position of the vertex considering positions of previous neighboring vertices in time.

[0091] In one implementation, the method further includes using a remesher to calculate the estimated mesh at the vertex and the particular time. In one implementation, the remesher uses the initial mesh as a template mesh to fit to the sequence of raw meshes to generate the sequence of estimated meshes. In one implementation, calculating the estimated mesh at the vertex and the particular time includes expressing the estimated mesh as Mt= maxp(Mt)), wherein Ot =Mtobservation data at the particular time, Mt= maximum a posteriori estimation of the estimated mesh at the particular time given Ot, M - = past smoothed meshes before the particular time, and N (v ) = connected vertex neighbors of the vertex. In one implementation, calculating the pure spatial part includes the positionAttorney Docket No. 113748-0313WO01of the vertex as close to the estimated mesh as possible. In one implementation, the method further includes executing an iterative form of a two-model smoother by maximizing the pure spatial part and the spatio-temporal part. In one implementation, the iterative form of the two- model smoother is expressed as

[0093] In one implementation, the iterative form of a two-model smoother is executed for all vertices and for all neighborhoods. In one implementation, estimating the position of the vertex considering positions of previous neighboring vertices in time includes smoothly changing shape of a neighborhood formed by the neighbors of the vertex along the estimated meshes calculated at previous in time to the particular time.

[0094] In another particular implementation, a method for temporal mesh smoothing is disclosed. The method includes: receiving an initial mesh and a sequence of raw meshes; calculating an estimated mesh at a vertex and a particular time using the initial mesh and the sequence of raw meshes; calculating a pure spatial part by calculating a conditional probability of the estimated mesh given a position of the vertex at the particular time; calculating a spatio-temporal part; and executing an iterative form of a two-model smoother by maximizing the pure spatial part and the spatio-temporal part.

[0095] In one implementation, calculating the estimated mesh includes calculating the estimated mesh as most likely mesh that best explains the sequence of raw meshes and neighbors of the vertex given estimated meshesAttorney Docket No. 113748-0313WO01calculated at previous in time to the particular time. In one implementation, calculating the spatio-temporal part includes estimating the position of the vertex considering positions of previous neighboring vertices in time. In one implementation, the iterative form of the two- model smoother is expressed as

[0097] In one implementation, the iterative form of a two-model smoother is executed for all vertices and for all neighborhoods. In one implementation, the temporal mesh smoothing is applied to digital characters. In one implementation, the digital characters comprise a talking head animation for TV programs or virtual front desks. In one implementation, the temporal mesh smoothing is applied to one of augmented-reality (AR), virtual-reality (VR), or extended-reality (XR) video generation. In one implementation, the video generation is made for commercials, games, or movies.

[0098] The description herein of the disclosed implementations is provided to enable any person skilled in the art to make or use the present disclosure.Numerous modifications to these implementations would be readily apparent to those skilled in the art, and the principals defined herein can be applied to other implementations without departing from the spirit or scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the implementations shown herein but is to be accorded the widest scope consistent with the principal and novel features disclosed herein.Attorney Docket No. 113748-0313WO01

[0099] Various implementations of the present disclosure are realized in electronic hardware, computer software, or combinations of these technologies. Some implementations include one or more computer programs executed by one or more computing devices. In general, the computing device includes one or more processors, one or more data-storage components (e. g., volatile or nonvolatile memory modules and persistent optical and magnetic storage devices, such as hard and floppy disk drives, CD-ROM drives, and magnetic tape drives), one or more input devices (e. g., game controllers, mice and keyboards), and one or more output devices (e. g., display devices).

[0100] The computer programs include executable code that is usually stored in a persistent storage medium and then copied into memory at run-time. At least one processor executes the code by retrieving program instructions from memory in a prescribed order. When executing the program code, the computer receives data from the input and / or storage devices, performs operations on the data, and then delivers the resulting data to the output and / or storage devices.

[0101] Those of skill in the art will appreciate that the various illustrative modules and method steps described herein can be implemented as electronic hardware, software, firmware or combinations of the foregoing. To clearly illustrate this interchangeability of hardware and software, various illustrative modules and method steps have been described herein generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the application and design constraints imposed on the overallAttorney Docket No. 113748-0313WO01system. Skilled persons can implement the described functionality in varying ways for each application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. In addition, the grouping of functions within a module or step is for ease of description. Specific functions can be moved from one module or step to another without departing from the present disclosure.

[0102] All features of each above-discussed example are not necessarily required in a particular implementation of the present disclosure. Further, it is to be understood that the description and drawings presented herein are representative of the subject matter that is broadly contemplated by the present disclosure. It is further understood that the scope of the present disclosure fully encompasses other implementations that may become obvious to those skilled in the art and that the scope of the present disclosure is accordingly limited by nothing other than the appended claims.

Claims

Attorney Docket No. 113748-0313WO01CLAIMS1. A method for temporal mesh smoothing, comprising: receiving an initial mesh and a sequence of raw meshes;calculating an estimated mesh at a vertex and a particular time as most likely mesh that best explains the sequence of raw meshes and neighbors of the vertex given estimated meshes calculated at previous in time to the particular time;calculating a pure spatial part by calculating a conditional probability of the estimated mesh given a position of the vertex at the particular time; and calculating a spatio-temporal part by estimating the position of the vertex considering positions of previous neighboring vertices in time.

2. The method of claim 1, further comprising using a remesher to calculate the estimated mesh at the vertex and the particular time.

3. The method of claim 2, wherein the remesher uses the initial mesh as a template mesh to fit to the sequence of raw meshes to generate the sequence of estimated meshes.

4. The method of claim 1, wherein calculating the estimated mesh at the vertex and the particular time includes expressing the estimated mesh as Mt=whereinOt = observation data at the particular time,Attorney Docket No. 113748-0313WO01Mt= maximum a posteriori estimation of the estimated mesh at the particular time given Ot,M - = past smoothed meshes before the particular time, andN (Vi) = connected vertex neighbors of the vertex.

5. The method of claim 1, wherein calculating the pure spatial part includes the position of the vertex as close to the estimated mesh as possible.

6. The method of claim 1, further comprising executing an iterative form of a two-model smoother by maximizing the pure spatial part and the spatiotemporal part.

7. The method of claim 6, wherein the iterative form of the two-model smoother is expressed as8. The method of claim 6, wherein the iterative form of the two-model smoother includes time and vertex variants < Jss and aTs and is expressed as9. The method of claim 6, wherein the iterative form of a two-model smoother is executed for all vertices and for all neighborhoods.Attorney Docket No. 113748-0313WO0110. The method of claim 1, wherein estimating the position of the vertex considering positions of previous neighboring vertices in time comprises smoothly changing shape of a neighborhood formed by the neighbors of the vertex along the estimated meshes calculated at previous in time to the particular time.

11. A method for temporal mesh smoothing, comprising:receiving an initial mesh and a sequence of raw meshes;calculating an estimated mesh at a vertex and a particular time using the initial mesh and the sequence of raw meshes;calculating a pure spatial part by calculating a conditional probability of the estimated mesh given a position of the vertex at the particular time;calculating a spatio-temporal part; andexecuting an iterative form of a two-model smoother by maximizing the pure spatial part and the spatiotemporal part.

12. The method of claim 11, wherein calculating the estimated mesh comprisescalculating the estimated mesh as most likely mesh that best explains the sequence of raw meshes and neighbors of the vertex given estimated meshes calculated at previous in time to the particular time.Attorney Docket No. 113748-0313WO0113. The method of claim 11, wherein calculating the spatio-temporal part comprisesestimating the position of the vertex considering positions of previous neighboring vertices in time.

14. The method of claim 11, wherein the iterative form of the two-model smoother is expressed as15. The method of claim 11, wherein the iterative form of the two-model smoother includes time and vertex variant σSs and σTs and is expressed as16. The method of claim 11, wherein the iterative form of a two-model smoother is executed for all vertices and for all neighborhoods.

17. The method of claim 11, wherein the temporal mesh smoothing is applied to digital characters.

18. The method of claim 17, wherein the digital characters comprise a talking head animation for TV programs or virtual front desks.

19. The method of claim 11, wherein the temporal mesh smoothing is applied to one of augmented reality (AR), virtual reality (VR), or extended reality (XR)Attorney Docket No. 113748-0313WO01video generation.

20. The method of claim 19, wherein the video generation is made for commercials, games, or movies.