Optimal and interactive keyframe selection for motion capture

Computational Visual Media, Apr 2019

Motion capture is increasingly used in games and movies, but often requires editing before it can be used, for many reasons. The motion may need to be adjusted to correctly interact with virtual objects or to fix problems that result from mapping the motion to a character of a different size or, beyond such technical requirements, directors can request stylistic changes. Unfortunately, editing is laborious because of the low-level representation of the data. While existing motion editing methods accomplish modest changes, larger edits can require the artist to “re-animate” the motion by manually selecting a subset of the frames as keyframes. In this paper, we automatically find sets of frames to serve as keyframes for editing the motion. We formulate the problem of selecting an optimal set of keyframes as a shortest-path problem, and solve it efficiently using dynamic programming. We create a new simplified animation by interpolating the found keyframes using a naive curve fitting technique. Our algorithm can simplify motion capture to around 10% of the original number of frames while retaining most of its detail. By simplifying animation with our algorithm, we realize a new approach to motion editing and stylization founded on the time-tested keyframe interface. We present results that show our algorithm outperforms both research algorithms and a leading commercial tool.

A PDF file should load here. If you do not see its contents the file may be temporarily unavailable at the journal website or you do not have a PDF plug-in installed and enabled in your browser.

Alternatively, you can download the file locally and open with any standalone PDF reader:

https://link.springer.com/content/pdf/10.1007%2Fs41095-019-0138-z.pdf

Optimal and interactive keyframe selection for motion capture

Computational Visual Media pp 1–21 | Cite as Optimal and interactive keyframe selection for motion capture AuthorsAuthors and affiliations Richard RobertsJ. P. LewisKen AnjyoJaewoo SeoYeongho Seol Open Access Research Article First Online: 13 April 2019 Abstract Motion capture is increasingly used in games and movies, but often requires editing before it can be used, for many reasons. The motion may need to be adjusted to correctly interact with virtual objects or to fix problems that result from mapping the motion to a character of a different size or, beyond such technical requirements, directors can request stylistic changes. Unfortunately, editing is laborious because of the low-level representation of the data. While existing motion editing methods accomplish modest changes, larger edits can require the artist to “re-animate” the motion by manually selecting a subset of the frames as keyframes. In this paper, we automatically find sets of frames to serve as keyframes for editing the motion. We formulate the problem of selecting an optimal set of keyframes as a shortest-path problem, and solve it efficiently using dynamic programming. We create a new simplified animation by interpolating the found keyframes using a naive curve fitting technique. Our algorithm can simplify motion capture to around 10% of the original number of frames while retaining most of its detail. By simplifying animation with our algorithm, we realize a new approach to motion editing and stylization founded on the time-tested keyframe interface. We present results that show our algorithm outperforms both research algorithms and a leading commercial tool. Keywordsmotion capture motion editing keyframe animation dynamic programming  Electronic Supplementary Material Supplementary material is available in the online version of this article at  https://doi.org/10.1007/s41095-019-0138-z. Richard Roberts researches into artist-directed tools for animation and visual effects work. Roberts is a currently a research fellow, developing a facial mocap and animation pipeline for the production of a VR narrative experience. He has worked briefly in industry, receiving credit in the Adventures of Tintin, and also has a background developing virtual machines for high-level programming languages. J. P. Lewis is a numerical programmer and researcher. He is principal research scientist at SEED, the new research lab of Electronic Arts, and is an adjunct associate professor in the machine learning group of Victoria University of Wellington. His interests include computer vision and machine learning applications in entertainment. He has received credits on a few movies including Avatar and the Matrix sequels, and several of his algorithms have been adopted in commercial software including Maya and MATLAB. Ken Anjyo set up and headed the research and development division of OLM Digital, the digital production company in Tokyo famous for the Pokémon movies and other 3D animated feature films. He became the company’s CTO and is now its executive R&D adviser. He is a board member of VFX-JAPAN, the Japanese association of domestic digital production companies, and a member of the Visual Effects Society. Since 2018, he has also been working as the director of the Computational Media Innovation Center at Victoria University of Wellington. Jaewoo Seo is a director of R&D at Pinscreen. His research interests include facial animation, motion capture, and GPU programming. Before joining Pinscreen, he was in the visual effects industry as an R&D engineer at ILM, Weta Digital, and OLM Digital. He received his Ph.D. and M.S. degrees in culture technology from KAIST and B.S. degree in digital media and in computer and information engineering from Ajou University. Yeongho Seol is interested in developing fundamental computer graphics and vision technologies and making them useful in real-world VFX and animation production. He is experienced in a range of motion capture related technologies and works as senior motion capture developer in Weta digital. Download to read the full article text Notes Acknowledgements Many researchers and artists have contributed important insights to this research. The authors would like to give special thanks to Ayumi Kimura and other staff of OLM Digital, to Johan Andersson, Ida Winterhaven, and Binh Le of SEED, Electronic Arts, and also to Ian Loh and other staff of Victoria University of Wellington’s Computational Media Innovation Centre and Virtual Worlds Lab. The authors would also like to thank the Moveshelf team for supporting the web-based presentation of our results. Supplementary material Supplementary material, approximately 145 MB. References [1] Lam, D. Personal communication. Electronic Arts, 2017.Google Scholar [2] Shelton, D. Personal communication. Electronic Arts, 2017.Google Scholar [3] White, T. Animation from Pencils to Pixels: Classical Techniques for Digital Animators. Burlington: Focal Press, 2006.Google Scholar [4] Roy, K. How to Cheat in Maya 2014: Tools and Techniques for Character Animation. Burlington: Focal Press, 2013.CrossRefGoogle Scholar [5] Ramer, U. An iterative procedure for the polygonal approximation of plane curves. Computer Graphics and Image Processing Vol. 1, No. 3, 244–256, 1972.CrossRefGoogle Scholar [6] Bertsekas, D. P. Network Optimization: Continuous and Discrete Models. Belmont: Athena Scientific, 1998.zbMATHGoogle Scholar [7] The U.S. game industry has 2,457 companies supporting 220,000 jobs. 2018. Available at https://venturebeat.com/2017/02/14/the-u-s-game-industry-has-2457-companies-supporting-220000-jobs. [8] The games industry in numbers. 2018. Available at https://https://ukie.org.uk. [9] Wang, X.; Chen, Q.; Wang, W. 3D human motion editing and synthesis: A survey. Computational and Mathematical Methods in Medicine Vol. 2014, Article ID 104535, 2014.Google Scholar [10] Miura, T.; Kaiga, T.; Shibata, T.; Katsura, H.; Tajima, K.; Tamamoto, H. A hybrid approach to keyframe extraction from motion capture data using curve simplification and principal component analysis. IEEJ Transactions on Electrical and Electronic Engineering Vol. 9, No. 6, 697–699, 2014.CrossRefGoogle Scholar [11] Wolin, A.; Eoff, B.; Hammond, T. ShortStraw: A simple and effective corner finder for polylines. In: Proceedings of the 5th Eurographics Conference on Sketch-based Interfaces and Modeling, 33–40, 2008.Google Scholar [12] So, C. K. F.; Baciu, G. Entropy-based motion extraction for motion capture animation. Computer Animation and Virtual Worlds Vol. 16, Nos. 3–4, 225–235, 2005.CrossRefGoogle Scholar [13] Cuntoor, N. P.; Chellappa, R. Key frame-based activity representation using antieigenvalues. In: Computer Vision — ACCV 2006. Lecture Notes in Computer Science, Vol. 3852. Narayanan, P. J.; Nayar, S. K.; Shum, H. Y. Eds. Springer Berlin Heidelberg, 499–508, 2006.Google Scholar [14] Wei, X. P.; Liu, R.; Zhang, Q. Key-frame extraction of human motion capture data based on least-square distance curve. Journal of Convergence Information Technology Vol. 7, 11–19, 2012.Google Scholar [15] Bulut, E.; Capin, T. Keyframe extraction from motion capture data by curve saliency. Available at http://www.people.vcu.edu/∼ebulut/casa.pdf. [16] Halit, C.; Capin, T. Multiscale motion saliency for keyframe extraction from motion capture sequences. Computer Animation and Virtual Worlds Vol. 22, No. 1, 3–14, 2011.CrossRefGoogle Scholar [17] Marr, D. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. New York: Henry Holt and Co., Inc., 1982.Google Scholar [18] Douglas, D.; Peucker, T. K. Algorithms for the reduction of the number of points required to represent a digitized line or its caricature. Cartographica: The International Journal for Geographic Information and Geovisualization Vol. 10, No. 2, 112–122, 1973.CrossRefGoogle Scholar [19] Lowe, D. G. Three-dimensional object recognition from single two-dimensional images. Artificial Intelligence Vol. 31, No. 3, 355–395, 1987.CrossRefGoogle Scholar [20] Lim, I. S.; Thalmann, D. Key-posture extraction out of human motion data. In: Proceedings of the 23rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Vol. 2, 1167–1169, 2001.Google Scholar [21] Seol, Y.; Seo, J.; Kim, P. H.; Lewis, J. P.; Noh, J. Artist friendly facial animation retargeting. ACM Transactions on Graphics Vol. 30, No. 6, Article No. 162, 2011.Google Scholar 22] Liu, X.-M.; Hao, A.-M.; Zhao, D. Optimization-based key frame extraction for motion capture animation. The Visual Computer Vol. 29, No. 1, 85–95, 2013.CrossRefGoogle Scholar [23] Zhang, Q.; Zhang, S.; Zhou, D. Keyframe extraction from human motion capture data based on a multiple population genetic algorithm. Symmetry Vol. 6, No. 4, 926–937, 2014.CrossRefGoogle Scholar [24] Zhang, Y.; Cao, J. 3D human motion key-frames extraction based on asynchronous learningfactor PSO. In: Proceedings of the 5th International Conference on Instrumentation and Measurement, Computer, Communication and Control, 1617–1620, 2015.Google Scholar [25] Chang, X.; Yi, P.; Zhang, Q. Key frames extraction from human motion capture data based on hybrid particle swarm optimization algorithm. In: Recent Developments in Intelligent Information and Database Systems. Studies in Computational Intelligence, Vol. 642. Król, D.; Madeyski, L.; Nguyen, N. Eds. Springer Cham, 335–342, 2016.Google Scholar [26] Bellman, R.; Kashef, B.; Vasudevan, R. Splines via dynamic programming. Journal of Mathematical Analysis and Applications Vol. 38, No. 2, 471–479, 1972.MathSciNetCrossRefzbMATHGoogle Scholar [27] Lewis, J. P.; Anjyo, K. Identifying salient points. In: Proceedings of the ACM SIGGRAPH ASIA 2009 Sketches, Article No. 41, 2009.Google Scholar [28] Witkin, A.; Popovic, Z. Motion warping. In: Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques, 105–108, 1995.Google Scholar [29] Lee, J.; Shin, S. Y. A hierarchical approach to interactive motion editing for human-like figures. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, 39–48, 1999.Google Scholar [30] Witkin, A.; Kass, M. Spacetime constraints. ACM SIGGRAPH Computer Graphics Vol. 22, No. 4, 159–168, 1988.CrossRefGoogle Scholar [31] Gleicher, M. Animation from observation: Motion capture and motion editing. ACM SIGGRAPH Computer Graphics Vol. 33, No. 4, 51–54, 1999.CrossRefGoogle Scholar [32] Guay, M.; Cani, M.-P.; Ronfard, R. The line of action: An intuitive interface for expressive character posing. ACM Transactions on Graphics Vol. 32, No. 6, Article No. 205, 2013.Google Scholar [33] Choi, B.; i Ribera, R. B.; Lewis, J. P.; Seol, Y.; Hong, S.; Eom, H.; Jung, S.; Noh, J. SketchiMo: Sketch-based motion editing for articulated characters. ACM Transactions on Graphics Vol. 35, No. 4, Article No. 146, 2016.Google Scholar [34] Kovar, L.; Gleicher, M.; Pighin, F. Motion graphs. ACM Transactions on Graphics Vol. 21, No. 3, 473–482, 2002.CrossRefGoogle Scholar [35] Casas, D.; Tejera, M.; Guillemaut, J.-Y.; Hilton, A. 4D parametric motion graphs for interactive animation. In: Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, 103–110, 2012.Google Scholar [36] Huang, P.; Tejera, M.; Collomosse, J.; Hilton, A. Hybrid skeletal-surface motion graphs for character animation from 4D performance capture. ACM Transactions on Graphics Vol. 34, No. 2, Article No. 17, 2015.Google Scholar [37] Park, M. J.; Shin, S. Y. Example-based motion cloning. Computer Animation and Virtual Worlds Vol. 15, Nos. 3–4, 245–257, 2004.CrossRefGoogle Scholar [38] Peng, X. B.; Abbeel, P.; Levine, S.; van de Panne, M. DeepMimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Transactions on Graphics Vol. 37, No.4, Article No. 143, 2018.Google Scholar [39] Assa, J.; Caspi, Y.; Cohen-Or, D. Action synopsis: Pose selection and illustration. ACM Transactions on Graphics Vol. 24, No. 3, 667–676, 2005.CrossRefGoogle Scholar [40] Yasuda, H.; Kaihara, R.; Saito, S.; Nakajima, M. Motion belts: Visualization of human motion data on a timeline. IEICE Transactions on Information and Systems Vol. E91-D, No. 4, 1159–1167, 2008.CrossRefGoogle Scholar [41] Hu, Y.; Wu, S.; Xia, S.; Fu, J.; Chen, W. Motion track: Visualizing variations of human motion data. In: Proceedings of the IEEE Pacific Visualization Symposium, 153–160, 2010.Google Scholar [42] Arikan, O. Compression of motion capture databases. ACM Transactions on Graphics Vol. 25, No. 3, 890–897, 2006.CrossRefzbMATHGoogle Scholar [43] Huang, K.-S.; Chang, C.-F.; Hsu, Y.-Y.; Yang, S.-N. Key probe: A technique for animation keyframe extraction. The Visual Computer Vol. 21, No. 8, 532–541, 2005.CrossRefGoogle Scholar [44] Tournier, M.; Wu, X.; Courty, N.; Arnaud, E.; Reveret, L. Motion compression using principal geodesics analysis. Computer Graphics Forum Vol. 28, No. 2, 355–364, 2009.CrossRefGoogle Scholar [45] Xia, G.; Sun, H.; Niu, X.; Zhang, G.; Feng, L. Keyframe extraction for human motion capture data based on joint kernel sparse representation. IEEE Transactions on Industrial Electronics Vol. 64, No. 2, 1589–1599, 2016.CrossRefGoogle Scholar [46] Dijkstra, E. W. A note on two problems in connexion with graphs. Numerische Mathematik Vol. 1, No. 1, 269–271, 1959.MathSciNetCrossRefzbMATHGoogle Scholar [47] Schneider, P. J. An algorithm for automatically fitting digitized curves. In: Graphics Gems. San Diego: Academic Press Professional, Inc., 612–626, 1990.CrossRefGoogle Scholar [48] Adobe Mixamo. 2018. Available at https://www.mixamo.com. [49] Kaufman, J. C.; Simonton, D. K. The Social Science of Cinema. Oxford University Press, 2013.Google Scholar [50] Miura, T.; Kaiga, T.; Katsura, H.; Tajima, K.; Shibata, T.; Tamamoto, H. Adaptive keypose extraction from motion capture data. Journal of Information Processing Vol. 22, No. 1, 67–75, 2014.CrossRefGoogle Scholar [51] Lasseter, J. Principles of traditional animation applied to 3D computer animation. ACM SIGGRAPH Computer Graphics Vol. 21, No. 4, 35–44, 1987.CrossRefGoogle Scholar Copyright information © The Author(s) 2019 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the articles Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj. Authors and Affiliations Richard Roberts1Email authorJ. P. Lewis2Ken Anjyo13Jaewoo Seo4Yeongho Seol51.Victoria University of WellingtonWellingtonNew Zealand2.SEED, Electronic ArtsLos AngelesUSA3.OLM DigitalTokyoJapan4.PinscreenLos AngelesUSA5.Weta DigitalWellingtonNew Zealand


This is a preview of a remote PDF: https://link.springer.com/content/pdf/10.1007%2Fs41095-019-0138-z.pdf

Richard Roberts, J. P. Lewis, Ken Anjyo, Jaewoo Seo, Yeongho Seol. Optimal and interactive keyframe selection for motion capture, Computational Visual Media, 2019, 1-21, DOI: 10.1007/s41095-019-0138-z