Bodies at Rest: 3D Human Pose and Shape Estimation From a Pressure Image Using Synthetic Data

CVPR, pp. 6214-6223, 2020.

Cited by: 1|Bibtex|Views80|DOI:https://doi.org/10.1109/CVPR42600.2020.00625
EI
Other Links: arxiv.org|dblp.uni-trier.de|academic.microsoft.com
Weibo:
We present the PressurePose dataset, a large-scale synthetic dataset consisting of 3D human body poses and shapes with pressure images

Abstract:

People spend a substantial part of their lives at rest in bed. 3D human pose and shape estimation for this activity would have numerous beneficial applications, yet line-of-sight perception is complicated by occlusion from bedding. Pressure sensing mats are a promising alternative, but training data is challenging to collect at scale. W...More

Code:

Data:

0
Introduction
  • Humans spend a large part of their lives resting. While resting, humans select poses that can be sustained with little physical exertion.
  • The lack of physical exertion and absence of motion makes this class of human activities amenable to relatively simple biomechanical models similar to the ragdoll models used in video games [39]
  • The authors apply this insight to the problem of using a pressure image to estimate the 3D human pose and shape of a person resting in bed.
  • The authors present PressureNet, a deep learning model that estimates 3D human body pose and shape from a low-resolution pressure image (Fig. 1, right)
Highlights
  • Humans spend a large part of their lives resting
  • The lack of physical exertion and absence of motion makes this class of human activities amenable to relatively simple biomechanical models similar to the ragdoll models used in video games [39]
  • We apply this insight to the problem of using a pressure image to estimate the 3D human pose and shape of a person resting in bed
  • We present the PressurePose dataset, a large-scale synthetic dataset consisting of 3D human body poses and shapes with pressure images (Fig. 1, left)
  • We present PressureNet, a deep learning model that estimates 3D human body pose and shape from a low-resolution pressure image (Fig. 1, right)
  • We found that our deep learning models would often make mistakes that neglected the role of contact between the body and the bed, such as placing the heel of a foot at a location some distance away from an isolated high pressure region
Methods
  • Participants donned an

    Optitrak motion capture suit with high contrast to the bed sheets to facilitate analysis of the pose and body shape.
  • The authors' released dataset consists of RGB images and depth/point cloud data from the Kinect that are gender limbs on bed train ct.
  • Real pose partition, limb distribution general*.
  • FN even leg space: {Y1, ...Y4} ∈ YL M N even arm space: {Y1, ...Y8} ∈ YA F Y.
  • FY even leg space, arms Fig. 10(b) M Y prone hands up†.
  • Even leg space, hnds above shldrs M Y supine crossed legs**.
  • Even leg space, even arm space, M N feet must cross according to x direction in Fig. 10(a) supine straight limbs**.
  • The authors' released dataset consists of RGB images and depth/point cloud data from the Kinect that are gender limbs on bed train ct. synth test ct. synth test ct. real pose partition, limb distribution general*
Results
  • The authors found that using more synthetic data resulted in higher performance in all tests, as shown in Table 2.
  • As expected, ablating the PMR network and ablating noise reduced performance.
  • Fig. 8 shows results from the best performing network with 184K training images, noise, and the PMR network.
  • The authors compared the error on a set of 99 participant selected poses, shown in Table 3, using the best performing PressureNet. Results show a higher error for lateral pos-
Conclusion
  • With the physics-based simulation pipeline, the authors generated a dataset, PressurePose, consisting of 200K synthetic pressure images with an unprecedented variety of body shapes and poses.
  • The authors trained a deep learning model, PressureNet, entirely on synthetic data.
  • With the best performing model, the authors achieve an average pose estimation error of < 5 cm, as measured by 3DVPE, resulting in accurate 3D pose and body shape estimation with real people on a pressure sensing bed
Summary
  • Introduction:

    Humans spend a large part of their lives resting. While resting, humans select poses that can be sustained with little physical exertion.
  • The lack of physical exertion and absence of motion makes this class of human activities amenable to relatively simple biomechanical models similar to the ragdoll models used in video games [39]
  • The authors apply this insight to the problem of using a pressure image to estimate the 3D human pose and shape of a person resting in bed.
  • The authors present PressureNet, a deep learning model that estimates 3D human body pose and shape from a low-resolution pressure image (Fig. 1, right)
  • Methods:

    Participants donned an

    Optitrak motion capture suit with high contrast to the bed sheets to facilitate analysis of the pose and body shape.
  • The authors' released dataset consists of RGB images and depth/point cloud data from the Kinect that are gender limbs on bed train ct.
  • Real pose partition, limb distribution general*.
  • FN even leg space: {Y1, ...Y4} ∈ YL M N even arm space: {Y1, ...Y8} ∈ YA F Y.
  • FY even leg space, arms Fig. 10(b) M Y prone hands up†.
  • Even leg space, hnds above shldrs M Y supine crossed legs**.
  • Even leg space, even arm space, M N feet must cross according to x direction in Fig. 10(a) supine straight limbs**.
  • The authors' released dataset consists of RGB images and depth/point cloud data from the Kinect that are gender limbs on bed train ct. synth test ct. synth test ct. real pose partition, limb distribution general*
  • Results:

    The authors found that using more synthetic data resulted in higher performance in all tests, as shown in Table 2.
  • As expected, ablating the PMR network and ablating noise reduced performance.
  • Fig. 8 shows results from the best performing network with 184K training images, noise, and the PMR network.
  • The authors compared the error on a set of 99 participant selected poses, shown in Table 3, using the best performing PressureNet. Results show a higher error for lateral pos-
  • Conclusion:

    With the physics-based simulation pipeline, the authors generated a dataset, PressurePose, consisting of 200K synthetic pressure images with an unprecedented variety of body shapes and poses.
  • The authors trained a deep learning model, PressureNet, entirely on synthetic data.
  • With the best performing model, the authors achieve an average pose estimation error of < 5 cm, as measured by 3DVPE, resulting in accurate 3D pose and body shape estimation with real people on a pressure sensing bed
Tables
  • Table1: Comparison of Literature: Human Pose in Bed
  • Table2: Results comparing testing data and network type
  • Table3: Results - participant selected poses. *See Fig. 9-top left
  • Table4: Partitions for synthetic data and prescribed poses. For evening the leg space, see Fig. 10(a). For evening the arm space, an additional four subspaces {Y5, . . . Y8} are chosen because the most distal joint (hand) is allowed to extend all the way below and above the limb root joint (shoulder), measured in the y direction
  • Table5: Partitioned results for prescribed poses with the best network for each real and synthetic
Download tables as Excel
Related work
  • Human pose estimation. There is long history of human pose estimation from camera images [2, 32, 41, 50, 51] and the more recent use of CNNs [53, 54]. The field has been moving rapidly with the estimation of 3D skeleton models [45, 59], and human pose and shape estimation as a 3D mesh [5, 29, 44] using human body models such as SCAPE [4] and SMPL [35]. These latter methods enforce physical constraints to provide kinematically feasible pose estimates, some via optimization [5] and others using learned embedded kinematics models [14, 29, 59]. Our approach builds on these works both directly through the use of available neural networks (e.g, SMPL embedding) and conceptually.

    While pressure image formation differs from conventional cameras, the images are visually interpretable and methods developed in the vision community are well suited to pressure imagery [9, 29, 54]. PressureNet’s model of pressure image generation relates to recent work on physical contact between people and objects [7, 25, 26]. It also data: (R)eal, (S)ynth modality:
Funding
  • This work was supported by the National Science Foundation Graduate Research Fellowship Program under Grant No DGE1148903, NSF award IIS-1514258, NSF award DGE1545287 and AWS Cloud Credits for Research
Reference
  • Felix Achilles, Alexandru-Eugen Ichim, Huseyin Coskun, Federico Tombari, Soheyl Noachtar, and Nassir Navab. Patient mocap: human pose estimation under blanket occlusion for hospital monitoring applications. In International Conference on Medical Image Computing and ComputerAssisted Intervention, pages 491–499. Springer, 2016. 3
    Google ScholarLocate open access versionFindings
  • Ankur Agarwal and Bill Triggs. Recovering 3d human pose from monocular images. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 28(1):44–58, 2006. 2
    Google ScholarLocate open access versionFindings
  • I. Akhter and M. J. Black. Pose-conditioned joint angle limits for 3d human pose reconstruction. In CVPR, 2015. 13
    Google ScholarLocate open access versionFindings
  • Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. SCAPE: shape completion and animation of people. ACM Transactions on Graphics, 24(3):408–416, 2005. 2
    Google ScholarLocate open access versionFindings
  • Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J. Black. Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image. In ECCV, pages 561–578. Springer, 2016. 2, 5, 10
    Google ScholarLocate open access versionFindings
  • Donna C. Boone and Stanley P. Azen. Normal range of motion of joints in male subjects. Journal of Bone and Joint Surgery, 61(5):756–759, 1979. 3, 13
    Google ScholarLocate open access versionFindings
  • Samarth Brahmbhatt, Cusuh Ham, Charles C. Kemp, and James Hays. ContactDB: Analyzing and predicting grasp contact via thermal imaging. In CVPR, pages 8709–8719. IEEE, 2019. 2
    Google ScholarLocate open access versionFindings
  • Adrian Bulat and Georgios Tzimiropoulos. Human pose estimation via convolutional part heatmap regression. In ECCV, pages 717–732, 2016. 3
    Google ScholarLocate open access versionFindings
  • Joao Carreira, Pulkit Agrawal, Katerina Fragkiadaki, and Jitendra Malik. Human pose estimation with iterative error feedback. In CVPR, pages 4733–4742. IEEE, 2016. 2, 3
    Google ScholarLocate open access versionFindings
  • Leslie Casas, Nassir Navab, and Stefanie Demirci. Patient 3d body pose estimation from pressure imaging. International Journal of Computer Assisted Radiology and Surgery, pages 1–8, 2019. 1, 2, 3
    Google ScholarLocate open access versionFindings
  • Kenny Chen, Paolo Gabriel, Abdulwahab Alasfour, Chenghao Gong, Werner K. Doyle, Orrin Devinsky, Daniel Friedman, Patricia Dugan, Lucia Melloni, Thomas Thesen, David Gonda, Shifteh Sattar, Sonya Wong, and Vikash Gilja. Patient-specific pose estimation in clinical environments. IEEE Journal of Translational Engineering in Health and Medicine, 6:1–11, 2018. 1, 3
    Google ScholarLocate open access versionFindings
  • Wenzheng Chen, Huan Wang, Yangyan Li, Hao Su, Zhenhua Wang, Changhe Tu, Dani Lischinski, Daniel Cohen-Or, and Baoquan Chen. Synthesizing training images for boosting human 3d pose estimation. In Conference in 3D Vision, pages 479–488. IEEE, 2016. 3
    Google ScholarLocate open access versionFindings
  • Alexander Clegg, Wenhao Yu, Zackory Erickson, Jie Tan, C. Karen Liu, and Greg Turk. Learning to navigate cloth using haptics. In International Conference on Intelligent Robots and Systems, pages 2799–2805. IEEE/RSJ, 2017. 3
    Google ScholarLocate open access versionFindings
  • Henry M. Clever, Ariel Kapusta, Daehyung Park, Zackory Erickson, Yash Chitalia, and Charles C. Kemp. 3D human pose estimation on a configurable bed from a pressure image. In IROS, pages 54–61. IEEE, 2018. 1, 2, 3, 6, 10, 13, 15, 16
    Google ScholarLocate open access versionFindings
  • Vandad Davoodnia, Saeed Ghorbani, and Ali Etemad. In-bed pressure-based pose estimation using image space representation learning. arXiv preprint arXiv:1908.08919, 2019. 3
    Findings
  • Zackory Erickson, Henry M. Clever, Greg Turk, C. Karen Liu, and Charles C. Kemp. Deep haptic model predictive control for robot-assisted dressing. In International Conference on Robotics and Automation (ICRA), pages 1–8. IEEE, 2018. 3
    Google ScholarLocate open access versionFindings
  • Zackory Erickson, Vamsee Gangaram, Ariel Kapusta, C. Karen Liu, and Charles C. Kemp. Assistive gym: A physics simulation framework for assistive robotics. In International Conference on Robotics and Automation (ICRA). IEEE, 2020. 3
    Google ScholarLocate open access versionFindings
  • M Farshbaf, Rasoul Yousefi, M Baran Pouyan, Sarah Ostadabbas, Mehrdad Nourani, and Matthew Pompeo. Detecting high-risk regions for pressure ulcer risk assessment. In BIBM, pages 255–260. IEEE, 2013. 1, 3
    Google ScholarLocate open access versionFindings
  • Robert Grimm, Sebastian Bauer, Johann Sukkau, Joachim Hornegger, and Gunther Greiner. Markerless estimation of patient orientation, posture and pose using range and pressure imaging. International journal of computer assisted radiology and surgery, 7(6):921–929, 2012. 1, 2, 3
    Google ScholarLocate open access versionFindings
  • Ahsan Habib, Isura Ranatunga, Kyle Shook, and Dan O. Popa. Skinsim: A simulation environment for multimodal robot skin. In International Conference on Automation Science and Engineering, pages 1226–1231, 2014. 3
    Google ScholarLocate open access versionFindings
  • Nikolaus Hansen, Youhei Akimoto, and Petr Baudis. CMA-ES/pycma on Github. Zenodo, DOI:10.5281/zenodo.2559634, Feb. 2019. 5
    Findings
  • Nikolaus Hansen, Sibylle D. Muller, and Petros Koumoutsakos. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (cmaes). Evolutionary Computation, 11(1):1–18, 2003. 11, 12
    Google ScholarLocate open access versionFindings
  • Tatsuya Harada, Taketoshi Mori, Yoshifumi Nishida, Tomohisa Yoshimi, and Tomomasa Sato. Body parts positions and posture estimation system based on pressure distribution image. In ICRA, volume 2, pages 968–975. IEEE, 1999. 3
    Google ScholarLocate open access versionFindings
  • Tatsuya Harada, Tomomasa Sato, and Taketoshi Mori. Pressure distribution image based human motion tracking system using skeleton and surface integration model. In ICRA, volume 4, pages 3201–3207. IEEE, 2001. 2, 3
    Google ScholarLocate open access versionFindings
  • Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael J. Black. Resolving 3d human pose ambiguities with 3d scene constraints. In ICCV. IEEE, 2019. 2
    Google ScholarFindings
  • Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J. Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In CVPR, pages 11807–11816, 2019. 2
    Google ScholarLocate open access versionFindings
  • Brayden Hollis, Stacy Patterson, Jinda Cui, and Jeff Trinkle. Bubbletouch: A quasi-static tactile skin simulator. In Artificial Intelligence for Human-Robot Interaction. AAAI, 2018. 3
    Google ScholarLocate open access versionFindings
  • Invacare. 5410IVC Full Electric Homecare Bed. www.invacare.com/cgi-bin/imhqprd/inv_
    Findings
  • catalog/prod_cat_detail.jsp?s=0&prodID= 5410IVC, Last accessed on 2020-02-27. 2
    Google ScholarFindings
  • [29] Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In CVPR, pages 7122 – 7131. IEEE, 2018. 2, 6, 14
    Google ScholarLocate open access versionFindings
  • [30] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. 15
    Google ScholarFindings
  • [31] Jeongseok Lee, Michael X. Grey, Sehoon Ha, Tobias Kunz, Sumit Jain, Yuting Ye, Siddhartha S. Srinivasa, Mike Stilman, and C. Karen Liu. Dart: Dynamic animation and robotics toolkit. Journal of Open Source Software, 3(22), 2018. 3, 4
    Google ScholarLocate open access versionFindings
  • [32] Jason J Liu, Ming-Chun Huang, Wenyao Xu, and Majid Sarrafzadeh. Bodypart localization for pressure ulcer prevention. In EMBC, pages 766–769. IEEE, 2014. 2, 3
    Google ScholarLocate open access versionFindings
  • [33] Shuangjun Liu and Sarah Ostadabbas. Seeing under the cover: A physics guided learning approach for in-bed pose estimation. In Medical Image Computing and Computer Assisted Intervention, pages 236–245. Springer, 2019. 3
    Google ScholarLocate open access versionFindings
  • [34] Shuangjun Liu, Yu Yin, and Sarah Ostadabbas. In-bed pose estimation: Deep learning with shallow dataset. IEEE Journal of Translational Engineering in Health and Medicine, 7:1–12, 2019. 3
    Google ScholarLocate open access versionFindings
  • [35] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. Smpl: A skinned multi-person linear model. ACM Transactions on Graphics, 34(6):248, 2015. 2, 3, 4
    Google ScholarLocate open access versionFindings
  • [36] Miles Macklin, Matthias Muller, Nuttapong Chentanez, and Tae-Yong Kim. Unified particle physics for real-time applications. ACM Transactions on Graphics, 33(4), 2014. 3,
    Google ScholarLocate open access versionFindings
  • https://github.com/MandyMo/pytorch HMR, 2018.14
    Findings
  • [38] Vista Medical. BodiTrak pressure mapping system solutions. www.boditrak.com/products/medical. php, Last accessed on 2020-02-27. 2
    Findings
  • [39] Ian Millington. Game physics engine development: how to build a robust commercial-grade physics engine for your game. CRC Press, 2010. 1
    Google ScholarFindings
  • [40] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In ECCV, pages
    Google ScholarLocate open access versionFindings
  • [41] Ryuzo Okada and Stefano Soatto. Relevant feature selection for human pose estimation and localization in cluttered images. In ECCV, pages 434–445. Springer, 2008. 2
    Google ScholarLocate open access versionFindings
  • [42] Sarah Ostadabbas, Maziyar Baran Pouyan, Mehrdad
    Google ScholarLocate open access versionFindings
  • [43] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017. 15
    Google ScholarFindings
  • [44] G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. Osman, D. Tzionas, and M. J. Black. Expressive body capture: 3d hands, face, and body from a single image. In CVPR, pages 10975–10985. IEEE, 2019. 2
    Google ScholarLocate open access versionFindings
  • [45] Georgios Pavlakos, Xiaowei Zhou, Konstantinos G. Derpanis, and Kostas Daniilidis. Coarse-to-fine volumetric prediction for single-image 3d human pose. In CVPR. IEEE, 2017.
    Google ScholarLocate open access versionFindings
  • [46] Shahram Payandeh and Naoufel Azouz. Finite elements, mass-spring-damper systems and haptic rendering. In ICRA, pages 224–229. IEEE. 5
    Google ScholarLocate open access versionFindings
  • [47] Zachary Pezzementi, Erica Jantho, Lucas Estrade, and Gregory D. Hager. Characterization and simulation of tactile sensors. In IEEE Haptics Symposium, pages 199–205, 2017. 3
    Google ScholarLocate open access versionFindings
  • [48] Anurag Ranjan, Javier Romero, and Michael J. Black. Learning human optical flow. In British Machine Vision Conference. BMVA Press, 2018. 4, 13
    Google ScholarLocate open access versionFindings
  • [49] Asbjørn Roaas and Gunnar BJ Andersson. Normal range of motion of the hip, knee and ankle joints in male subjects, 3040 years of age. Acta Orthopaedica Scandinavica, 53(2):205–208, 1982. 3, 13
    Google ScholarLocate open access versionFindings
  • [50] Nikolaos Sarafianos, Bogdan Boteanu, Catalin Ionescu, and Ioannis A. Kakadiaris. 3d human pose estimation: A review of the literature and analysis of covariates. Computer Vision and Image Understanding, (152):1–20, 2016. 2
    Google ScholarLocate open access versionFindings
  • [51] Jamie Shotton, Andrew Fitzgibbon, Mat Cook, Toby Sharp, Mark Finocchio, Richard Moore, Alex Kipman, and Andrew Blake. Real-time human pose recognition in parts from single depth images. In CVPR, pages 1297–1304, 2011. 2
    Google ScholarLocate open access versionFindings
  • [52] J. M. Soucie, C. Wang, A. Forsyth, S. Funk, M. Denny, K. E. Roach, D. Boone, and Hemophilia Treatment Center Network. Range of motion measurements: reference values and a database for comparison studies. Haemophilia, 17(3):500– 507, 2011. 3, 13
    Google ScholarLocate open access versionFindings
  • [53] Jonathan J. Tompson, Arjun Jain, Yann LeCun, and Christoph Bregler. Joint training of a convolutional network and a graphical model for human pose estimation. In Advances in neural information processing systems, pages 1799–1807, 2014. 2
    Google ScholarLocate open access versionFindings
  • [54] Alexander Toshev and Christian Szegedy. Deeppose: Human pose estimation via deep neural networks. In CVPR, pages 1653–1660, 2014. 2, 3
    Google ScholarLocate open access versionFindings
  • [55] Aydin Tozeren. Human Body Dynamics: Classical Mechanics and Human Movement. Springer, 1999. 5, 10
    Google ScholarFindings
  • [56] Gul Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J. Black, Ivan Laptev, and Cordelia Schmid. Learning from synthetic humans. In CVPR, pages 109–117, 2017. 3
    Google ScholarLocate open access versionFindings
  • [57] Thiemo Wiedemeyer. IAI Kinect2. https://github.com/code-iai/iai_kinect2, 2014 – 2015. Accessed June 12, 2015.12
    Findings
  • [58] Tao Yu, Zerong Zheng, Yuan Zhong, Jianhui Zhao, Qionghai Dai, Gerard Pons-Moll, and Yebin Liu. Simulcap: Singleview human performance capture with cloth simulation. In CVPR, pages 5499–5509. IEEE, 2019. 3
    Google ScholarLocate open access versionFindings
  • [59] Xingyi Zhou, Xiao Sun, Wei Zhang, Shuang Liang, and Yichen Wei. Deep kinematic pose regression. In ECCV 2016 Workshops, pages 186–201, 2016. 2
    Google ScholarLocate open access versionFindings
Full Text
Your rating :
0

 

Tags
Comments