Browsing by Person "Balch-Tomes, Jonathan"

Now showing 1 - 2 of 2

Beyond the edge: Markerless pose estimation of speech articulators from ultrasound and camera images using DeepLabCut
(MDPI, 2022-02-02) Wrench, Alan A.; Balch-Tomes, Jonathan
Automatic feature extraction from images of speech articulators is currently achieved by detecting edges. Here, we investigate the use of pose estimation deep neural nets with transfer learning to perform markerless estimation of speech articulator keypoints using only a few hundred hand-labelled images as training input. Midsagittal ultrasound images of the tongue, jaw, and hyoid and camera images of the lips were hand-labelled with keypoints, trained using DeepLabCut and evaluated on unseen speakers and systems. Tongue surface contours interpolated from estimated and hand-labelled keypoints produced an average mean sum of distances (MSD) of 0.93, s.d. 0.46 mm, compared with 0.96, s.d. 0.39 mm, for two human labellers, and 2.3, s.d. 1.5 mm, for the best performing edge detection algorithm. A pilot set of simultaneous electromagnetic articulography (EMA) and ultrasound recordings demonstrated partial correlation among three physical sensor positions and the corresponding estimated keypoints and requires further investigation. The accuracy of the estimating lip aperture from a camera video was high, with a mean MSD of 0.70, s.d. 0.56, mm compared with 0.57, s.d. 0.48 mm for two human labellers. DeepLabCut was found to be a fast, accurate and fully automatic method of providing unique kinematic data for tongue, hyoid, jaw, and lips.
Improving ultrasound post estimation accuracy by training on co-registered EMA data
(University of Aizu, 2024-06-28) Balch-Tomes, Jonathan; Wrench, Alan A.; Scobbie, James M.; Macmartin, C.; Turk, A.
This study aims to assess how accurately DeepLabCut [1], when applied to ultrasound tongue images, can estimate Electromagnetic Articulography (EMA) sensor positions. EMA provides objective measures of anterior tongue, jaw, and lip kinematics. DeepLabCut pose estimation is a powerful method of extracting keypoint positions from midsagittal ultrasound images of the tongue. It has an advantage over EMA in that it can be applied to the whole of the tongue from tip to root as well as the jaw and the hyoid. After correction for probe translation standard error in the estimation of keypoint positions compared to the corresponding EMA sensor positions was 1.2-1.5mm along the tongue contour and 0.5-0.9mm perpendicular to the tongue contour.