[Deep learning-assisted construction of three-demensional facial midsagittal plane].

objectiveTo establish a deep learning algorithm that can accurately determine three-dimensional facial anatomical landmarks, multi-view stacked hourglass convolutional neural networks (MSH-CNN) and to construct three-dimensional facial midsagittal plane automatically based on MSH-CNN and weighted Procrustes analysis algorithm.

methodsOne hundred subjects with no obvious facial deformity were collected in our oral clinic. Three-dimensional facial data were scanned by three-dimensional facial scanner. Experts annotated twenty-one facial landmarks and midsagittal plane of each data. Eighty three-dimensional facial data were used as training set, to train the MSH-CNN in this study. The overview of MSH-CNN network architecture contained multi-view rendering and training the MSH-CNN network. The three-dimensional facial data were rendered from ninety-six views that were fed to MSH-CNN and the output was one heatmap per landmark. The result of the twenty-one landmarks was accurately placed on the three-dimensional facial data after a three-dimensional view ray voting process. The remaining twenty three-dimensional facial data were used as test set. The trained MSH-CNN automatically determined twenty-one three-dimensional facial anatomical landmarks of each case of data, and calculated the distance between each MSH-CNN landmark and the expert landmark, which was defined as position error. The midsagittal plane of the twenty subjects' could be automatically constructed, using the MSH-CNN and Procrustes analysis algorithm. To evaluate the effect of midsagittal plane by automatic method, the angle between the midsagittal plane constructed by the automatic method and the expert annotated plane was calculated, which was defined as angle error.

resultsFor twenty subjects with no obvious facial deformity, the average angle error of the midsagittal plane constructed by MSH-CNN and weighted Procrustes analysis algorithm was 0.73°±0.50°, in which the average position error of the twenty-one facial landmarks automatically determined by MSH-CNN was (1.13±0.24) mm, the maximum position error of the orbital area was (1.31±0.54) mm, and the minimum position error of the nasal area was (0.79±0.36) mm.

conclusionThis research combines deep learning algorithms and Procrustes analysis algorithms to realize the fully automated construction of the three-dimensional midsagittal plane, which initially achieves the construction effect of clinical experts. The obtained results constituted the basis for the independent intellectual property software development.