Analysis device and analysis method

Abstract

This analysis device (10) acquires event data output from an event camera (20) that captures an image of a field of view including a contact position between an object (60) and an object-hitting element (50). The analysis device (10) is configured to estimate a contact position of the object (60) and the object-hitting element (50) with respect to the object (60) or the object-hitting element (50) on the basis of the acquired event data, and output information indicating the estimated contact position.

Description

Analysis device and analysis method

[0001] The present disclosure relates to an analysis device and an analysis method.

[0002] There is a need to measure the impact of a ball by a sports implement (striking implement) such as a racket, a club, or a bat from the aspects of implement design or user guidance. For example, a technique for calculating the contact position (impact point) of a sports implement and a ball by image analysis using an image captured by a camera is known.

[0003] For example, Japanese Patent No. 7072590 (Patent Document 1) describes a technique for obtaining a first image and a second image captured within a predetermined time of an impact between an object and an element among a plurality of images captured by an imaging device at a constant frame rate, and specifying the impact position (impact point) of the object. In Patent Document 1, in the first image and the second image, the object is discriminated, and the impact position can be determined by image processing for detecting an impact from the movement of the object.

[0004] Japanese Patent No. 7072590

[0005] In Patent Document 1, mainly impact point detection in golf is exemplified, and it is stated that the imaging device needs to have a frame rate high enough to acquire a plurality of frames of a golf swing before and after an impact.

[0006] Moreover, not limited to golf, generally, when there is a contact (impact) between an "object striking element" such as a striking implement or a part of the human body (foot, hand, etc.) and an "object" such as a ball, since the object striking element and the object move at high speed, there is a concern that motion blur may occur in a captured image by a general frame camera.

[0007] Therefore, from the viewpoint of improving the analysis accuracy of the impact point (contact position), it is preferable to increase the frame rate of image capturing by using a so-called high-speed camera.

[0008] However, increasing the frame rate necessitates processing large amounts of image data, which increases the processing load due to the increased amount of data and computation required for temporary data storage and retrieval of captured data, as well as image processing. As a result, there are concerns that the longer processing time will make it difficult to perform immediate analysis of impact points (contact locations). Furthermore, because high-speed cameras have short exposure times, they cannot capture images in low-light environments, raising concerns about limitations on the shooting environment.

[0009] This disclosure was made to solve these problems, and the purpose of this disclosure is to analyze the object impact element and the object's contact position from the captured data while suppressing the processing load.

[0010] In certain aspects of this disclosure, an analysis device is provided. The analysis device comprises a data acquisition unit, an estimation processing unit, and a result output unit. The data acquisition unit is configured to acquire event data output from an event camera that captures a field of view including the contact positions of an object and an object impacting element. The estimation processing unit is configured to estimate the contact positions of an object and an object impacting element with respect to the object or object impacting element based on the event data. The result output unit is configured to output information indicating the contact positions estimated by the estimation processing unit.

[0011] In another aspect of this disclosure, an analysis method is provided. The analysis method comprises: (1) acquiring event data output from an event camera that captures a field of view including the contact positions of an object and an object striking element; (2) estimating the contact positions of the object and the object striking element with respect to the object or object striking element based on the event data; and (3) outputting information indicating the estimated contact positions.

[0012] According to this disclosure, by using event data from an event camera in which each pixel in the field of view asynchronously detects an event when the brightness changes, the processing load for estimating the object impact element and the object's contact position can be reduced, and the object impact element and the object's contact position can be analyzed from the captured data.

[0013] This is a conceptual diagram illustrating an example of a dot analysis system using the analysis device according to this embodiment. This is a conceptual diagram illustrating the field of view of the event camera. This is a conceptual diagram showing a side view of Figure 2 before and after impact, viewed from direction III. This is a block diagram illustrating an example of the hardware configuration of the analysis device according to this embodiment. This is a diagram illustrating the structure of event data from the event camera. This is a functional block diagram of the analysis device according to this embodiment. This is a diagram illustrating the structure of event data packets. This is a conceptual diagram showing a first example of output information from the result output unit shown in Figure 6. This is a conceptual diagram showing a second example of output information from the result output unit shown in Figure 6. This is a flowchart illustrating the analysis method according to Embodiment 1 executed by the analysis device. This is a flowchart illustrating the details of the impact time estimation process (S200) in Figure 9. This is a conceptual diagram illustrating the definition of the amount of event data used for estimating the impact time. This is a conceptual diagram illustrating the processing content in S210 of Figure 10. This is a conceptual diagram illustrating the processing content in S220 of Figure 10. This is a flowchart illustrating the details of the image processing (S300) in Figure 9. This is a conceptual diagram for illustrating the image of an event image. This is a conceptual diagram illustrating the image of the binarization process for an event image. This is a conceptual diagram illustrating the image of the convex hull process for an event image. This is a conceptual diagram illustrating the image of convex hull processing on an event image. This is a conceptual diagram illustrating the image of ellipse discrimination processing on an event image. This is a conceptual diagram illustrating the image of the event image obtained in S310 of Figure 14. This is a conceptual diagram illustrating the image of the event image obtained in S320 of Figure 14. This is a conceptual diagram illustrating the image of the event image obtained in S330 of Figure 14. This is a conceptual diagram illustrating the image of the event image obtained in S340 of Figure 14. This is a conceptual diagram illustrating the image of the impact image obtained in S350 of Figure 14. This is a flowchart illustrating the analysis method according to Embodiment 2 executed by the analysis device. This is a first conceptual diagram illustrating the types of event images obtained each period. This is a second conceptual diagram illustrating the types of event images obtained each period. This is a third conceptual diagram illustrating the types of event images obtained each period. This is a fourth conceptual diagram illustrating the types of event images obtained each period.This is the fifth conceptual diagram explaining the types of event images obtained at each cycle.

[0014] Embodiments of this disclosure will be described in detail below with reference to the drawings. In the following, the same or corresponding parts in the drawings will be denoted by the same reference numerals, and their descriptions will not be repeated in principle.

[0015] Embodiment 1. <Configuration> Figure 1 is a conceptual diagram illustrating a dot analysis system using an analysis device 10, which is a representative example of the analysis device according to this embodiment.

[0016] The analysis device 10 uses the racket 50 as an example of an "object striking element" and the ball 60 as an example of an "object," and when the player 2 swings the racket 50 and impacts the ball 60, it detects the point of impact of the ball 60 on the surface of the racket 50 (racket face) as the "contact position" of the object striking element and the object.

[0017] As shown in Figure 1, event data from an event camera 20 equipped with an event-based vision sensor (EVS), or in addition to this, audio data detected by a microphone 30, is input to the analysis device 10 via the communication means 40. The microphone 30 corresponds to one embodiment of a "trigger generator device" that can be used to estimate the impact time, and the audio data corresponds to one embodiment of "trigger data".

[0018] The analysis device 10 can typically be configured using a PC (Personal Computer), but it may also be a tablet device, smartphone, virtual machine in a cloud environment, server, or any other information processing device.

[0019] Figure 2 is a conceptual diagram illustrating the field of view of the event camera 20. As shown in Figure 2, the racket 50 is swung by the player 2 (Figure 1) along the trajectories indicated by arrows 51 (before impact) and 52 (after impact), while the event camera 20 is positioned so that its field of view 11 includes the entire racket 50 when it impacts the ball 60.

[0020] For example, the event camera 20 can be positioned to capture the impact from the front or back, adjusted so that the player 2, the racket 50, and the ball 60 do not overlap within the field of view 11.

[0021] Figure 3 is a conceptual diagram showing a side view of Figure 2, viewed from direction III, before and after impact.

[0022] In Figure 3, the position of the racket 50 at the moment of impact after being swung along arrow 51 is shown, and after impact it swings further along arrow 52. In contrast, the ball 60 approaches the racket 50 in a trajectory along arrow 61 until impact, and at the moment of impact it momentarily comes to rest on the racket face before moving along arrow 62 in the opposite direction to arrow 61.

[0023] Thus, it is understood that while the racket 50 moves continuously in the same direction before and after impact, the ball 60 moves in the opposite direction before and after impact, and also stops instantaneously at the moment of impact.

[0024] Referring again to Figure 1, the microphone 30 can be positioned to detect the sound of the ball hitting the racket at impact, which serves as a trigger for estimating the impact time. The communication means 40 may consist of either wired communication via a communication line or wireless communication. The analysis device 10 detects the impact point using at least one of the event data from the event camera 20 and the audio data (trigger data) from the microphone 30, which are acquired via the communication means 40. Note that the "trigger device" for providing the trigger for estimating the impact time is not limited to the microphone 30, but can also be an acceleration sensor or pressure sensor that measures the acceleration or pressure of the racket 50 or ball 60, a photoelectric sensor that includes the impact point in its projection range, and a manual switch operated by an observer.

[0025] Figure 4 shows an example of the hardware configuration of the analysis device 10 according to this embodiment. Although Figure 4 illustrates a hardware configuration assuming a PC, as described above, it is also possible to configure the analysis device 10 using equipment other than a PC.

[0026] Referring to Figure 4, the analysis device 10 includes, as its main components, a CPU (Central Processing Unit) 102, a memory 104, a touch panel 106, a button 108, a display 110, a wireless communication unit 112, a communication antenna 113, a memory interface (I / F) 114, a speaker 116, a microphone 118, a communication interface (I / F) 120, and a camera 122. The recording medium 115 is an external storage medium.

[0027] The CPU 102 controls the operation of each part of the analysis device 10 by reading and executing a program stored in the memory 104. More specifically, the CPU 102 implements each of the processes (steps) of the analysis device 10, which will be described later, by executing the program.

[0028] Memory 104 is implemented using RAM (Random Access Memory), ROM (Read-Only Memory), flash memory, etc. Memory 104 stores programs executed by the CPU 102, or data used by the CPU 102.

[0029] The touch panel 106 is provided on the display 110, which functions as a display unit, and may be of any type, such as resistive or capacitive. The button 108 is located on the surface of the analysis device 10 and receives instructions from the user and inputs those instructions to the CPU 102. The display 110 is located on the surface of the analysis device 10. In some cases, it may be used in connection with the analysis device 10. In that case, the analysis device 10 is provided with an output interface for connecting the display 110. In other cases, the display 110 may be any output device such as a cathode ray tube display, a liquid crystal display, or an organic EL (Electro-Luminescence) display.

[0030] The wireless communication unit 112 connects to a mobile communication network via the communication antenna 113 and transmits and receives signals for wireless communication. This enables the analysis device 10 to communicate with a predetermined external device via a mobile communication network such as LTE (Long Term Evolution).

[0031] The memory interface (I / F) 114 can read data from the external recording medium 115. The CPU 102 can read data stored on the external recording medium 115 via the memory interface 114 and store that data in the memory 104. The CPU 102 can read data from the memory 104 and store that data in the external recording medium 115 via the memory interface 114.

[0032] The recording medium 115 may include a CD (Compact Disc), DVD (Digital Versatile Disc), BD (Blu-ray® Disc), USB (Universal Serial Bus) memory, memory card, FD (Flexible Disk), or a hard disk, or other non-volatile media for storing programs.

[0033] The speaker 116 outputs sound based on instructions from the CPU 102. The microphone 118 receives speech directed to the analysis device 10. In Figure 1, if the analysis device 10 can be placed near the user (player) 2, the microphone 118 may be used instead of the microphone 30 to acquire audio data as "trigger data".

[0034] The communication interface (I / F) 120 is a communication interface for sending and receiving data between the analysis device 10 and external devices, and is implemented by adapters, connectors, etc. The communication method may be wired or wireless, and for wireless communication, BLE (Bluetooth Low Energy: registered trademark) or wireless LAN (Local Area Network) can be used. The CPU 102 can acquire event data from the event camera 20 and audio data from the microphone 30 using the communication I / F 120. The analysis device 10 may also acquire this data via a communication network such as the Internet.

[0035] Here, unlike a normal camera and a high-speed camera which capture images using luminance data of all pixels corresponding to the field of view 11 in synchronization with a constant frame rate, the event camera 20 is configured to detect an event when each of the multiple pixels within the field of view 11 exceeds a threshold luminance change.

[0036] Figure 5 shows a diagram illustrating the structure of event data from the event camera 20. Whenever a brightness change exceeding a threshold occurs in any of the multiple pixels covering the field of view 11, the event camera 20 detects the event and generates the event data Devn shown in Figure 5. Since the generation of event data by the event camera 20 is a common technique, a detailed explanation is omitted. In other words, the event camera 20 used in the analysis apparatus according to this embodiment does not need to be special; a general-purpose camera can be used.

[0037] The event data Devn includes the position data (x, y coordinates within the field of view 11) of the pixel where the event occurred, polarity data indicating whether the brightness change is "+" (increase in brightness) or "-" (decrease in brightness), and time data (t) indicating when the event occurred. The event camera 20 is configured to generate event data asynchronously each time an event is detected at each pixel.

[0038] Therefore, the event camera 20 has a time resolution [s] for event detection that is equivalent to the shutter speed, which is expressed as the reciprocal of the frame rate in a high-speed camera, in microseconds (10 －６ (s)) Order can be secured. On the other hand, the amount of output data per second (amount of event data) of the event camera 20 is generally suppressed to about 1 to 100 MB, compared to the order of 10 GB for a high-speed camera. <Data analysis processing> Figure 6 is a functional block diagram of the analysis device 10 according to this embodiment.

[0039] Referring to Figure 6, the analysis device 10 comprises a data acquisition unit 130, an estimation processing unit 140, and a result output unit 150.

[0040] The data acquisition unit 130 acquires event data Devn from the event camera 20. At this time, the event data may be transmitted from the event camera 20 to the analysis device 10 in the form of an event packet as illustrated in Figure 7.

[0041] Referring to Figure 7, an event packet 15 is a collection of multiple event data Devn. Each time a certain number of event data Devn are accumulated, one event packet 15 is generated and transmitted from the event camera 20 to the analysis device 10. Therefore, the amount of data in the event packets 15 transmitted within a certain time period corresponds to the amount of event data, which is the number of events that occurred.

[0042] Referring again to Figure 6, the data acquisition unit 130 may further acquire audio data Daud from the microphone 30.

[0043] Based on the data acquired by the data acquisition unit 130, the estimation processing unit 140 estimates the hitting point of the ball 60 by the racket 50, that is, the impact point on the racket surface. The impact point corresponds to an example of the "contact part" between the racket 50, which is an example of the "object hitting element", and the ball 60, which is an example of the "object". In Embodiment 1, the impact point is estimated by image processing using the event data Devn, which will be described later.

[0044] For each impact caused by one swing, the estimation processing unit 140 outputs impact point coordinate data Dimp indicating the coordinates (x, y) of the impact point in the x - y plane including the racket surface based on the event data. The coordinates of the impact point are indicated by the relative position of the contact part of the ball 60 (object) with respect to the racket 50 (object hitting element).

[0045] The estimation processing unit 140 includes a contact time estimation unit 142 and an imaging processing unit 144. The contact time estimation unit 142 calculates an estimated impact time, which is an example of the "estimated contact time", using at least one of the event data Devn from the event camera 20 and the audio data Dau at the microphone 30 during shooting by the event camera 20. The imaging processing unit 144 creates an impact image based on the event image using the event data in a time interval including the estimated impact time. As will be described later, the coordinates (x, y) of the impact point are calculated based on the detected positions of the racket 50 and the ball 60 in the impact image.

[0046] Based on the impact point coordinate data Dimp from the estimation processing unit 140, the result output unit 150 generates information to be output to the user (player 2) and outputs the output information to the display 110 or the like.

[0047] FIGS. 8A and 8B show examples of output information from the result output unit 150. In the first example shown in FIG. 8A, a plurality of impact points 70 in each of a plurality of swings are superimposed and displayed in the x-y plane including the racket face. Thus, each impact point coordinate data Dimp can be represented by coordinates (x, y) in the x-y plane normalized such that the full length in the lateral direction of the racket face is in the range of -100 (%) ≤ x ≤ +100 (%), and the full length in the longitudinal direction of the racket face is in the range of -100 (%) ≤ y ≤ +100 (%).

[0048] In the second example of FIG. 8B, an example of output information is shown in the form of a heat map in which the presence density for each region is color-coded for the plurality of impact points 70 in FIG. 8A. Thus, using the impact point coordinate data Dimp obtained by the estimation processing unit 140 for each swing, output information for the user (player 2) can be generated.

[0049] Note that the data indicating the output information may be transmitted to a server (not shown) or the like. For example, from the server, it is also possible to send back to the analysis device 10 diagnostic information or the like obtained by further processing (e.g., comparison with other users) the output information transmitted from the analysis device 10. That is, the output of information by the result output unit 150 is not limited to output by an image, and also includes outputting the data itself.

[0050] <Analysis Processing of Event Data According to Embodiment 1> Next, the analysis processing of event data according to Embodiment 1 will be described in detail.

[0051] FIG. 9 is a flowchart for explaining the analysis method according to Embodiment 1 executed by the analysis device 10. Each step in each flowchart described below can be basically realized by software processing in which the CPU 102 executes a program installed in advance in the analysis device 10. However, for some steps, or for some processing of each step, it is also possible to realize them by hardware processing using a dedicated electronic circuit, logic circuit, or the like provided in the analysis device 10.

[0052] Referring to Figure 9, the analysis device 10 (CPU 102) acquires event data Devn generated by the event camera 20 in step (hereinafter simply referred to as "S") 100. Thus, for example, when the event camera 20 captures one swing motion including one impact, the event data Devn output from the event camera 20 is sequentially input to the analysis device 10. The collection of input event data Devn is temporarily stored inside the analysis device 10 using memory 104 (Figure 2), etc. The processing in S100 realizes the function of the data acquisition unit 130 (Figure 6). In S100, audio data from the microphone 30, which is one embodiment of "trigger data" from the "trigger device," may also be acquired.

[0053] In S200, the analysis device 10 performs an impact time estimation process. In Embodiment 1, the estimated impact time is calculated based on the amount of event data transmitted from the event camera 20 or the audio data from the microphone 30.

[0054] In S300, the analysis device 10 performs image processing of the event data Devn for the time interval including the estimated impact time calculated in S200. Furthermore, in S400, the analysis device 10 estimates the impact point using the impact image obtained in S300. As a result, in S400, the impact point coordinate data Dimp in Figure 6 is generated. The processing from S200 to S400 realizes the function of the estimation processing unit 140 (Figure 6), and in particular, the processing in S200 realizes the function of the "contact time estimation unit", and the processing in S300 realizes the function of the "image processing unit".

[0055] Next, we will further explain the details of each process from S200 to S400. Figure 10 is a flowchart illustrating the details of the impact time estimation process (S200) in Figure 9. Figure 10 illustrates an example of a process that estimates the impact time based on the amount of event data transmitted from the event camera 20.

[0056] As shown in Figure 10, S200 in Figure 9 includes S210 and S220. In S210, the analysis device 10 extracts candidate impact intervals on the time axis, including the impact time, based on the amount of event data from the event camera 20. Furthermore, in S220, it calculates one point on the time axis among the candidate impact intervals extracted in S210 as the estimated impact time.

[0057] Figure 11 shows a conceptual diagram illustrating the definition of the amount of event data used to estimate the impact time.

[0058] Referring to Figure 11, the time intervals for calculating the amount of event data are set on the time axis with a period Ts(s). At the i-th calculation time t[i] from the beginning, the total amount of data transmitted from the event camera 20 in the interval from t[i]-Tw(s) to t[i]+Tw(s), defined by a fixed time window Tw(s), is set as the event data amount Q[i] at that time t[i]. Similarly, for each Ts interval, the event data amount Q is calculated in the interval defined by the time window Tw(s).

[0059] Furthermore, at each time point t[i], the difference from the event data amount Q[i-1] at the previous data amount calculation time t[i-1] is calculated as the event data difference amount DQ[i] (DQ[i] = Q[i] - Q[i-1]). For example, the period Ts for calculating the data amount can be about 0.5 [ms], and the time window Tw can be about 0.2 (ms).

[0060] Figures 12 and 13 show conceptual diagrams illustrating the processing steps in S210 and S220 of Figure 10, respectively.

[0061] As shown in Figure 12, when the event camera 20 starts capturing images, the number of events detected at each pixel of the event camera 20 increases as the racket 50 and ball 60 enter the field of view 11 as described in Figure 2. As a result, the amount of event data Q increases, and after both are fully in the field of view, the amount of event data Q decreases.

[0062] Therefore, the time t1 at which the event data amount Q reaches its maximum value at the start of shooting is set as the start of the impact candidate interval Tpc. Furthermore, the measurement time t2 at which the event data amount Q decreases to the maximum value α (%) (0 < α < 100) is set as the end of the impact candidate interval Tpc. For example, α can be set to approximately 80 (%).

[0063] As a result, in S210, the period from t1 to t2 in Figure 12 is defined as the impact candidate interval Tpc based on the time progression of the event data amount Q.

[0064] As shown in Figure 13, within the candidate impact interval (t1 to t2), the time at which the event data difference amount DQ is minimized is identified as the estimated impact time tp.

[0065] As explained in Figure 3, within the impact candidate interval, it is assumed that the racket 50 continues to move, while the ball 60 stops moving at the moment of impact. Therefore, at the moment of impact, it is expected that the number of events corresponding to the movement of the racket 50 remains unchanged, while the number of events corresponding to the movement of the ball 60 decreases sharply.

[0066] Therefore, at the moment of impact, the timing when the event data difference amount DQ is minimized (maximum absolute value when DQ < 0) can be estimated as the estimated impact time tp. In this way, according to S210 and S220, the estimated impact time can be calculated based on the amount of event data output from the event camera 20.

[0067] The impact time estimation process using S200 is not limited to estimation based on the temporal changes in the amount of event data as described above; other methods can also be applied.

[0068] For example, it is possible to calculate the estimated impact time based on the contrast of events. Specifically, for each event data (event packet) that occurs at regular time intervals, a histogram of each pixel is obtained, the variance across all pixels is calculated and compiled into a time series, and the time at which the maximum or minimum value of the variance or its difference in the time series occurs, or the time at which a threshold is exceeded, can be determined as the estimated impact time.

[0069] Alternatively, the estimated impact time can be determined by frequency analysis (short-time Fourier transform, wavelet transform, etc.) using event packets or event images divided into fixed time intervals as input, and then determining the time at which the maximum or minimum value of the frequency or its difference occurs, or the time at which a threshold is exceeded.

[0070] Alternatively, the impact time estimation process in S200 may be performed using a machine learning model that takes event data acquired during a single impact as input and outputs an estimated impact time, based on a machine learning model that uses the actual impact time as training data for a set of event data acquired during impact imaging.

[0071] In this way, based on the characteristics of the event data output asynchronously from each pixel when an event is detected, the impact time can be estimated from the event data using various methods.

[0072] In cases where audio data from microphone 30 is input to analysis device 10, the estimated impact time may be calculated based on the timing of the detection of the impact sound (i.e., based only on "trigger data"), without using event data. Alternatively, it is possible to determine the estimated impact time using both event data and audio data by combining the processing in S210 and S220 (event data amount basis), etc., with processing based on audio data. Furthermore, as mentioned above, output data from an acceleration sensor, pressure sensor, photocell, or manual switch can be used as "trigger data".

[0073] Referring again to Figure 9, in Embodiment 1, the analysis device 10 performs image processing (S300) of event data for a time interval including the estimated impact time in S200 (S220), and impact point estimation processing (S400) based on the impact image obtained in S300.

[0074] Figure 14 shows a flowchart illustrating the details of the image processing (S300) shown in Figure 9.

[0075] Referring to Figure 14, S300 for image processing includes S310 to S340. The analysis device 10 generates event images at the estimated impact time in S200 in S310, S330, and S340, respectively. The event image is generated by an image processing that accumulates event data for a specified cumulative time before and after the reference time (here, the estimated impact time), including the reference time. S300 for image processing further includes S350. In S350, an impact image is generated.

[0076] In S310, an event image for cropping the racket portion is obtained by imaging the accumulated time T0.

[0077] In contrast, in S330, the cumulative time is set to T1 (T1 < T0), which is shorter than T0, and an event image is acquired to detect the racket face at the time of impact. Then, in S340, the cumulative time is set to T2 (T1 < T2 < T0), which is longer than T1, and an event image is acquired to detect the ball at the time of impact. For example, T0 can be set to approximately 2000 (μs), T1 to approximately 500 (μs), and T2 to approximately 1000 (μs).

[0078] Figure 15 shows a conceptual diagram to explain the image of an event image obtained by accumulating event data.

[0079] Referring to Figure 15, the event image is obtained by imaging the event data for the accumulated times T0, T1, and T2 described above. It is obtained by drawing the set of pixels where an event occurred. For example, the set of pixels where an event with polarity data = "1 (increased brightness)" was detected can be drawn as a white region, while the set of pixels where an event with polarity data = "0 (decreased brightness)" was detected can be imaged as a blue region.

[0080] The event image 80 obtained in this way shows that the racket 50 and the ball 60 moved from the position in the blue area to the position in the white area during the cumulative period.

[0081] In the conceptual diagram of Figure 15, an example of an event image 80 that was actually obtained shows the outer edges of the white areas 81 (racket) and 85 (ball) with solid lines, while the outer edges of the blue areas 82 (racket) and 85 (ball) with dotted lines.

[0082] Image processing as illustrated in Figures 16 to 18 can be performed on the event images obtained in this way.

[0083] Figure 16 shows a conceptual diagram illustrating the process of binarization applied to event images.

[0084] As shown in Figure 16, the white and blue regions can be extracted from the event image 80 (Figure 15) by binarization. This allows for the determination of candidate regions for the objects (racket 50 and ball 60).

[0085] Figures 17A and 17B show conceptual diagrams illustrating the concept of convex hull processing for event images.

[0086] As shown in Figure 17A, the image obtained by the binarization process in Figure 16 shows image loss in the frame contour of the racket 50. Therefore, the frame contour of the racket can be treated as a convex set, and correction can be performed by applying a convex hull. This is expected to correct the frame contour as shown in Figure 17B.

[0087] Figure 18 shows a conceptual diagram illustrating the ellipse discrimination process applied to event images.

[0088] As shown in Figure 18, using a shape detection process such as the Hough transform, the ellipse 91 corresponding to the racket position can be identified from the outline shape 81z of the racket 50 in the processed event images of Figures 16, 17A, and 17B. Similarly, an ellipse 92 that is close to a circle and corresponds to the ball position can also be identified using the same shape detection process.

[0089] By applying the image processing described in Figures 16 to 18 to the event images obtained from the event data, it becomes possible to estimate the impact point based on the position detection of the racket 50 and ball 60 within the event image.

[0090] Referring again to Figure 14, in S310, the analysis device 10 applies the binarization process described in Figure 16 to the event image at the accumulated time T0, and also performs a trimming process on the outline of the racket portion. Furthermore, in S320, a frame correction process for detecting the racket face is performed on the image obtained in S310. Such trimming processing can improve the processing accuracy of the subsequent S330 to S350. In other words, the trimming processing in S310 and S320 can be omitted, but it is preferable to perform it in order to improve accuracy.

[0091] Figures 19A and 19B are conceptual diagrams illustrating the image of the event image obtained in S310 and S320, respectively.

[0092] Figure 19A shows the event image 80x obtained after the trimming process in S310, which includes the trimmed white region 81 and blue region 82. However, in region 95 of the event image 80x, there is a missing portion where the frame contour is interrupted.

[0093] Figure 19B shows the event image 80y after frame correction processing in S320. In the event image 80y, it can be seen that the missing portion in Figure 19A has been corrected within region 95 by applying the convex hull processing described in Figures 17A and 17B. As a result, the continuous frame contour of the racket 50 can be obtained.

[0094] Referring again to Figure 14, in S330, the analysis device 10 uses the event image at the cumulative time T1 as a candidate image to detect a candidate position of the racket face at the time of impact, and in S340, it uses the event image at the cumulative time T2 (T2 > T1) as a candidate image to detect a candidate position of the ball 60. Furthermore, in S350, the analysis device 10 generates an impact image using the candidate position of the racket face obtained in S330 and the candidate position of the ball 60 obtained in S340.

[0095] Figures 20A and 20B are conceptual diagrams illustrating the images of event images (candidate images) obtained in S330 and S340, respectively.

[0096] In Figure 20A, the event image 80a acquired in S330 with an accumulated time T1 is shown as a candidate image. On the other hand, in Figure 20B, the event image 80b acquired in S340 with an accumulated time T2 is shown as a candidate image.

[0097] In the event image (candidate image) 80a (Figure 20A), an ellipse 91a corresponding to the racket face and an ellipse 92a corresponding to the ball 60 are shown, obtained by the ellipse discrimination process described in Figure 18. Similarly, in the event image (candidate image) 80b (Figure 20B), an ellipse 91b corresponding to the racket face and an ellipse 92b corresponding to the ball 60 are also shown, obtained by the ellipse discrimination process.

[0098] Comparing Figures 20A and 20B, it can be seen that, due to the difference in accumulated time T1 and T2, ellipse 91a is closer to the shape of the racket face (racket surface) than ellipse 91a. On the other hand, between ellipses 91b and 92b, ellipse 92b is closer to the shape of the ball (perfect circle) than ellipse 91b.

[0099] Regarding the ball 60, in the event image 80a (Figure 20A) with a short integration time (T1), the position of the ball 60 cannot be captured sharply, whereas in the event image 80b (Figure 20B) with a long integration time (T2), the position of the ball 60 can be grasped as a shape close to a circle. In other words, for detecting the position of the ball 60, a certain amount of longer integration time is advantageous.

[0100] Conversely, with respect to the racket 50, it can be seen that the shape of the racket face is detected more accurately in the ellipse 91a in the event image 80a (Figure 20A) at the cumulative time T1 than in the ellipse 91b in the event image 80b (Figure 20B) at the cumulative time T2, where the effect of motion blur is evident. In other words, for detecting the position of the racket 50, a shorter cumulative time is advantageous compared to detecting the position of the ball 60.

[0101] Figure 21 shows a conceptual diagram illustrating the image of the impact image obtained in S350 of Figure 14.

[0102] As shown in Figure 21, the impact image 80f can be obtained by combining the background image (the part other than the ball) of the event image 80a in S330 with the ellipse 92b (Figure 20B) from the event image 80b in S340. That is, the impact image 80f can be generated without generating a new image, based on the already created event image 80a (Figure 20A) and event image 80b (Figure 20B), without increasing the processing load. The ellipse 91a from Figure 20A corresponds to one embodiment of the "detection position" of the racket 50 (object striking element), and the ellipse 92b from Figure 20B corresponds to one embodiment of the "detection position" of the ball 60 (object).

[0103] In S400 (Figure 9), the analysis device 10 can define the coordinate system (x-y plane) of the racket face using the racket position indicated by the ellipse 91a in the impact image 80f. In this case, it is preferable to define the minor axis direction of the ellipse 91a as x and the major axis direction as y.

[0104] Furthermore, the center point of the ellipse 92b indicating the ball position in the impact image 80f can be defined as the impact point (contact position) in the racket face coordinate system (x-y plane). The impact point can be represented by the coordinates (x, y) in the x-y plane as shown in Figure 8A, where -100 (%) ≤ x ≤ +100 (%) and -100 (%) ≤ y ≤ +100 (%), as illustrated in Figure 21.

[0105] Referring again to Figure 9, in S500, the analysis device 10 performs processing and / or output processing of the impact point estimation results obtained in S400. As illustrated in Figures 8A and 8B, it is also possible to perform the processing in S500 on data for multiple impact points 70. The processing in S500 realizes the function of the result output unit 150 (Figure 6).

[0106] As described above, in the analysis device according to this embodiment, the impact point, i.e., the contact position between the two, can be estimated by using event data from the event camera 20, in which each pixel asynchronously detects an event, to generate an impact image, and by image processing including position recognition (detection) of the racket 50 (object hitting element) and ball 60 (object) on the event image.

[0107] Therefore, compared to image processing of images captured at high frame rates using high-speed cameras, etc., for identifying impact points, the amount of processing data can be significantly reduced. Furthermore, while reducing the amount of computation, the impact point (contact position) can be estimated from the captured data with the same high accuracy as a high-speed camera. In addition, since known processing methods such as those described in Figures 15 to 17A and 17B can be arbitrarily used as image processing when creating event images, the accuracy of impact point estimation can be improved.

[0108] Furthermore, according to the event data analysis process described in Embodiment 1, the impact point is estimated based on the event image at a single point in time, based on the estimated impact time, thus reducing the amount of computation. In this way, by reducing the amount of processed data and computation, the effect of reducing the processing load for estimating the contact position of the object hitting element (e.g., racket) and the object (e.g., ball) can be enhanced.

[0109] Embodiment 2. Embodiment 2 describes in detail another example of the event data analysis process.

[0110] Figure 22 is a flowchart illustrating the analysis method according to Embodiment 2, which is performed by the analysis device 10. The processing of each step shown in Figure 22 can, in principle, be implemented in the analysis device 10 by software processing, where the CPU 102 executes a pre-installed program. Furthermore, some steps, or some processing of each step, can be implemented by hardware processing using dedicated electronic circuits or logic circuits provided within the analysis device 10.

[0111] Referring to Figure 22, the analysis device 10 acquires event data Devn generated by the event camera 20 using S100, similar to that in Figure 9.

[0112] In Embodiment 2, the analysis device 10 extracts multiple impact candidate images included within the impact interval by processing S600, which includes S610 to S630.

[0113] The analysis device 10 generates an event image by imaging the event data at intervals of period Tx(s) in S610. For example, Tx can be set to approximately 500 (μs). The processing in S610 can be equivalent to the processing in S300 for generating a single impact image in Embodiment 1. That is, in Embodiment 2, S610 generates an image corresponding to Figure 21, which shows the ellipses 91a and 92b obtained after the various processing described in Embodiment 1.

[0114] In this embodiment 2, by appropriately combining position correction using a Kalman filter or a particle filter among multiple event images, the position recognition accuracy in the process of acquiring candidate images for identifying (recognizing) the positions of the racket 50 and the ball 60 can be improved.

[0115] Furthermore, the analysis device 10 performs the judgment processes in S620 and S630 on the event images generated in S610 for each period Tx to extract candidate impact images.

[0116] Figures 23A to 23E are conceptual diagrams illustrating the types of event images obtained at each period Tx in S610. The event images obtained at each period Tx change in the time series shown in Figures 23A to 23E.

[0117] Figure 23A shows an event image obtained when neither the racket 50 nor the ball 60 is present within the field of view 11 of the event camera 20.

[0118] In S620 of Figure 22, it is detected whether or not object positions (elliptical shapes) corresponding to the racket 50 and ball 60 are detected in the event image generated in S610. For the type of event image shown in Figure 23A, S620 is determined to be NO, and the process returns to S610, generating the event image for the next cycle.

[0119] Figure 23B shows an event image obtained outside the impact interval in which the positions of the racket 50 and the ball 60 (elliptical shapes) are detected within the event image, but there is no overlap between their positions.

[0120] In contrast, Figure 23C shows the event image obtained at the beginning of the impact section, Figure 23D shows the moment of impact, and Figure 23E shows the event image obtained at the end of the impact section. In the impact section shown in Figures 23C to 23E, it can be seen that there is an overlap between the object positions (elliptical shapes) corresponding to the racket 50 and the ball 60 within the event image.

[0121] Furthermore, after the impact section ends, as shown in Figure 23E, there will again be no overlap between the object positions corresponding to the racket 50 and the ball 60 in the event image, similar to Figure 23B.

[0122] In S630 of Figure 22, it is determined whether the event image is within the impact interval (Figures 23C to 23E) by checking whether there is any overlap between the object positions corresponding to the racket 50 and the ball 60 (ellipses 91a and 92b in Figure 22) within the event image generated in S610. For the event image outside the impact interval in Figure 23B, S630 returns a NO result, and the process returns to S610, generating the event image for the next cycle.

[0123] In S700, the analysis device 10 processes multiple event images within the impact interval for which S630 was determined to be YES, treating them as multiple impact candidate images, and then performs a process to determine one impact image from among the multiple impact candidate images. S700 includes S710 to S730.

[0124] In step S710, the analysis device 10 performs an impact time estimation process. For example, similar to steps S210 and S220 in Figure 10, the estimated impact time can be calculated by determining the event data amount Q and the event data difference amount DQ at the reference time of the event image generated in step S610.

[0125] Then, in S720, the analysis device 10 determines whether or not the estimated impact time has arrived, based on the impact time estimation process in S710. For example, in light of the behavior of the event data difference amount DQ immediately before and after the estimated impact time tp shown in Figure 13, in the period when the event data difference amount DQ changes from decreasing to increasing, S720 can be judged as YES, and it can be determined that the impact time has arrived.

[0126] On the other hand, within the interval where the event data difference amount DQ continues to decrease (Figure 23C), S720 is judged as NO and the process returns to S610, generating the event image for the next cycle.

[0127] If the analysis device 10 determines S720 to be YES, S730 can determine the event image from the previous cycle as the impact image corresponding to the estimated impact time in S710.

[0128] Furthermore, the impact time estimation process and impact time arrival determination in S710 and S720 can be modified in the same way as described in S200 of Embodiment 1. Alternatively, the processing in S710 and S720 may be performed to identify the time when the event data difference amount DQ is at its minimum value after calculating the event data difference amount DQ corresponding to all of the multiple event images within the impact interval.

[0129] Furthermore, in Embodiment 2, it is also possible to perform the determinations in S710 and S720 based on the amount of change in the position of the ball 60 (ellipse 92b). As described above, at the moment of impact, the ball 60 momentarily stops, so the amount of change in the position of the ball 60 (ellipse 92b) decreases compared to the section immediately before impact. In other words, instead of the event data difference amount DQ, it is possible to calculate the estimated impact time by corresponding it to a decrease in the parameter value used to evaluate the amount of change in the position of the ball 60 (ellipse 92b).

[0130] Furthermore, as a unique determination made possible in Embodiment 2, it is also possible to use the velocity vector of the ball 60 (ellipse 92b) after position detection between multiple generated event images (within the impact interval) to calculate the estimated impact time in S710 and S720, corresponding to the velocity reversal timing.

[0131] The analysis device 10 can perform the same processing as in Figure 9, S400 and S500, on the impact image determined in S730. As a result, in S500, processing and / or output processing of the impact point estimation result obtained in S700 (S730) can be performed. As a result, the same function as the result output unit 150 (Figure 6) in Embodiment 1 can be realized, generating the output information exemplified in Figures 8A and 8B, and outputting it to the user.

[0132] In the analysis method according to Embodiment 2 shown in Figure 22, the functions of the "image processing unit" are realized by the processing in S610 and S730, and the functions of the "contact time estimation unit" are realized by the processing in S710 and S720.

[0133] As described above, in Embodiment 2, by performing image processing equivalent to the impact image in Embodiment 1 on multiple impact candidate images, the likelihood that the final impact image correctly captures the positions of the racket 50 and ball 60 at the time of impact increases. Furthermore, by appropriately combining the position correction using the Kalman filter and particle filter described above, the accuracy of position recognition of the racket 50 and ball 60 can be improved. As a result, it is expected that the accuracy of estimating the impact point (contact position) can be improved compared to Embodiment 1.

[0134] On the other hand, in Embodiment 2, image processing equivalent to the impact image in Embodiment 1 is performed multiple times, so the amount of computational processing increases compared to Embodiment 1. However, as with Embodiment 1, the amount of processed data and computational processing is suppressed compared to image processing on images captured at a high frame rate using a high-speed camera, etc., so it is understood that the processing load for estimating the contact position of the object impact element (e.g., racket) and the object (e.g., ball) can be suppressed.

[0135] In this embodiment, an analysis device and method for estimating the point of impact of the ball 60 on the racket 50 during the swing of the racket 50 as the "contact position" have been illustrated. However, the combination of the "object striking element" corresponding to the racket 50 and the "object" corresponding to the ball 60 is not limited to this example.

[0136] For example, the racket 50 may be replaced with any "striking tool" for striking a ball, such as a golf club, a bat, or a racket other than a tennis racket, as the "object striking element," thereby realizing the analysis apparatus and analysis method according to this embodiment.

[0137] Alternatively, in sports such as soccer, rugby, and volleyball, where an object (ball) is struck with the hands or feet of the human body, the player's hands or feet may be used as the "object striking element" instead of the racket 50. Furthermore, the "object" can be anything other than a ball, such as a shuttlecock in badminton, the tip of a sword in fencing, the ground in long jump, or the target of a strike (including the opponent) in boxing, karate, etc. By performing the above-described analysis process on event data capturing contact between any "object striking element" and "object," including these examples, the analysis device and analysis method according to this embodiment can be realized.

[0138] Thus, the analysis apparatus and analysis method according to this disclosure can be commonly applied to the estimation of the contact position of an "object" with respect to an "object striking element" when any "object striking element" and "object" come into contact. In this embodiment, which exemplifies a racket 50 and a ball 60, it is preferable that the integration time T1 of the first event image (80a) for detecting the position of the "object striking element" is longer than the integration time T2 of the second event image (80b) for detecting the position of the "object". However, depending on the "object striking element" and "object" to be analyzed, the integration time T1 for detecting the position of the "object striking element" and the integration time T2 for detecting the position of the "object" can be set to different values ​​suitable for their respective position detections, and the length / shortness of the integration times T1 and T2 may be the opposite of those in this embodiment (Figures 14, 20A, 20B, and 21).

[0139] In this embodiment, since the example described was one in which the racket 50 is larger than the ball 60, the impact point (contact position) was calculated as the relative position of the ball 60's contact point with the racket face. On the other hand, in sports such as rugby and soccer, where a "striking element" contacts a part of an "object" equivalent to a ball, the impact point can be calculated as the relative position of the contact point of the foot or other object with respect to the ball.

[0140] Furthermore, although this embodiment describes an example of estimating the impact point (the contact position between the object and the object striking element) when the ball 60 (object) is in motion in time prior to impact, the analysis device and analysis method according to this disclosure can also be applied in common when the ball 60 (object) is stationary in time prior to impact. In such cases, contrary to the examples in Figures 12 and 13, it is assumed that the amount of event data increases at the time of impact, but similarly, the estimated impact time can be determined based on the event data difference amount DQ, that is, the amount of change in event data.

[0141] In addition, in each flowchart described in this embodiment, the execution order of each step may be rearranged to the extent that is theoretically possible, or multiple steps may be executed in parallel to the extent that the processing load is acceptable. Alternatively, a group of processes may be divided into multiple executions at intervals of a certain amount of event data.

[0142] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the invention is indicated by the claims rather than by the foregoing description, and all modifications within the meaning and scope of the claims are intended to be included.

[0143] 2 Player, 10 Analysis device, 11 Field of view (event camera), 15 Event packet, 20 Event camera, 30 Microphone, 40 Communication means, 50 Racket, 60 Ball, 70 Impact point, 80, 80a, 80b, 80x, 80y Event image, 80f (80a') Impact image, 81 White area, 81z Contour shape, 82 Blue area, 91, 91a, 91b, 92, 92a, 92b Ellipse (detection position), 95 Area, 104 Memory, 106 Touch panel, 108 Button, 110 Display, 112 Wireless communication unit, 113 Communication antenna, 114 Memory interface, 115 Recording medium, 116 Speaker, 118 Microphone (analysis device), 122 Camera, 130 Data acquisition unit, 140 Estimation processing unit, 150 Result output unit, Q Event data amount, DQ Event data difference amount, Daud audio data, Devn event data, Dimp impact point coordinate data, T0, T1, T2 cumulative time, Tpc impact candidate interval, Ts, Tx period, Tw time window, tp estimated impact time.

Claims

1. An analysis device comprising: a data acquisition unit configured to acquire event data output from an event camera that captures a field of view including the contact positions of an object and an object impacting element; an estimation processing unit configured to estimate the contact positions of the object and the object impacting element with respect to the object or the object impacting element based on the event data; and a result output unit configured to output information indicating the contact positions estimated by the estimation processing unit.

2. The analysis apparatus according to claim 1, wherein the estimation processing unit is configured to calculate the estimated contact time of the object and the object impact element based on at least one of the event data and trigger data obtained by a trigger generating device when the event camera is taking photographs, and to calculate the coordinates of the contact position based on the detected positions of the object and the object impact element in an impact image based on an event image using the event data for a time interval including the estimated contact time.

3. The analysis apparatus according to claim 2, wherein the estimation processing unit includes a contact time estimation unit that calculates an estimated contact time of the object and the object impact element based on at least one of the event data and the trigger data, and an imaging processing unit that creates the impact image based on an event image using the event data for a time interval including the estimated contact time.

4. The analysis apparatus according to claim 2, wherein the estimation processing unit includes an imaging processing unit that generates a plurality of event images using the event data for a plurality of time intervals each containing a plurality of time points, and extracts a plurality of impact candidate images from the plurality of event images based on the detection positions of the object and the object impact element in each of the event images, and a contact time estimation unit that calculates the estimated contact time of the object and the object impact element based on at least one of the event data and the trigger data, and the imaging processing unit further determines the impact image from the plurality of impact candidate images based on the estimated contact time.

5. The analysis apparatus according to claim 3, wherein the impact image comprises the detection position of the object impact element in the event image where the time interval is a first time length, and the detection position of the object in the event image where the time interval is a second time length different from the first time length.

6. The analysis apparatus according to claim 4, wherein each of the plurality of event images includes the detection position of the object striking element in the event image where the time interval is a first time length, and the detection position of the object in the event image where the time interval is a second time length different from the first time length.

7. The analysis apparatus according to claim 3 or 4, wherein the contact time estimation unit calculates the amount of event data generated in the interval before and after each of a plurality of time intervals, and calculates the estimated contact time based on the amount of change in the amount of data from the previous time at each of the plurality of time intervals.

8. The analysis apparatus according to claim 4, wherein the contact time estimation unit calculates the estimated contact time based on the amount of change in position or the change in velocity vector of the object or the object impacting element between the plurality of impact candidate images.

9. An analysis method comprising: acquiring event data output from an event camera that captures a field of view including the contact positions of an object and an object striking element; estimating the contact positions of the object and the object striking element with respect to the object or the object striking element based on the event data; and outputting information indicating the estimated contact positions.

10. The analysis method according to claim 9, wherein estimating the contact position includes calculating an estimated contact time of the object and the object impact element based on at least one of the event data and trigger data obtained by a trigger generating device when the event camera is capturing images; and calculating the coordinates of the contact position based on the detected positions of the object and the object impact element in an impact image based on an event image using the event data for a time interval including the estimated contact time.

11. The analysis method according to claim 10, further comprising estimating the contact position by performing an imaging process to create the impact image based on an event image using the event data for a time interval including the estimated contact time.

12. The analysis method according to claim 10, wherein the estimation of the contact position includes generating a plurality of event images using the event data for a plurality of time intervals each containing a plurality of time points; performing an imaging process to extract a plurality of impact candidate images from the plurality of event images based on the detection positions of the object and the object impact element in each of the event images; calculating an estimated contact time of the object and the object impact element based on at least one of the event data and the trigger data; and identifying the impact image from the plurality of impact candidate images based on the estimated contact time.

13. The analysis method according to claim 11, wherein the impact image comprises the detection position of the object impact element in the event image where the time interval is a first time length, and the detection position of the object in the event image where the time interval is a second time length different from the first time length.

14. The analysis method according to claim 12, wherein each of the plurality of event images comprises the detection position of the object striking element in the event image where the time interval is a first time length, and the detection position of the object in the event image where the time interval is a second time length different from the first time length.

15. The analysis method according to any one of claims 10 to 12, wherein calculating the estimated contact time includes calculating the amount of event data that occurred in the interval before and after each of a plurality of time intervals, and calculating the estimated contact time based on the amount of change in the amount of data from the previous time in each of the plurality of time intervals.

Images