Internal computational models of physical bodies are fundamental to the ability of robots and animals alike to plan and control their actions. These “self-models” allow robots to consider outcomes of multiple possible future actions without trying them out in physical reality. Recent progress in fully data-driven self-modeling has enabled machines to learn their own forward kinematics directly from task-agnostic interaction data. However, forward kinematic models can only predict limited aspects of the morphology, such as the position of end effectors or velocity of joints and masses. A key challenge is to model the entire morphology and kinematics without prior knowledge of what aspects of the morphology will be relevant to future tasks. Here, we propose that instead of directly modeling forward kinematics, a more useful form of self-modeling is one that could answer space occupancy queries, conditioned on the robot’s state. Such query-driven self-models are continuous in the spatial domain, memory efficient, fully differentiable, and kinematic aware and can be used across a broader range of tasks. In physical experiments, we demonstrate how a visual self-model is accurate to about 1% of the workspace, enabling the robot to perform various motion planning and control tasks. Visual self-modeling can also allow the robot to detect, localize, and recover from real-world damage, leading to improved machine resiliency.

Get full access to this article

View all available purchase options and get full access to this article.

Already a subscriber or AAAS Member? Log In

Supplementary Materials

This PDF file includes:

Supplementary Results
Tables S1 and S2
Supplementary Methods

Other Supplementary Material for this manuscript includes the following:

Movies S1 to S6


G. G. Gallup Jr., Self-awareness and the emergence of mind in primates. Am. J. Primatol. 2, 237–248 (1982).
P. Rochat, Five levels of self-awareness as they unfold early in life. Conscious. Cogn. 12, 717–731 (2003).
N. Koenig, A. Howard, Design and use paradigms for gazebo, an open-source multi-robot simulator, in 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)(IEEE Cat. No. 04CH37566, IEEE, 2004), vol. 3, pp. 2149–2154.
E. Todorov, T. Erez, Y. Tassa, Mujoco: A physics engine for model-based control, in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE, 2012), pp. 5026–5033.
F. Faure, C. Duriez, H. Delingette, J. Allard, B. Gilles, S. Marchesseau, H. Talbot, H. Courtecuisse, G. Bousquet, I. Peterlik, in SOFA: A Multi-Model Framework for Interactive Physical Simulation, Soft Tissue Biomechanical Modeling for Computer Assisted Surgery (Springer, 2012), pp. 283–321.
E. Coumans, Y. Bai, PyBullet, A Python Module for Physics Simulation for Games, Robotics, and Machine Learning;
J. Lee, M. X. Grey, S. Ha, T. Kunz, S. Jain, Y. Ye, S. S. Srinivasa, M. Stilman, C. Karen Liu, Dart: Dynamic animation and robotics toolkit. J. Open Source Softw. 3, 500 (2018).
R. Tedrake, Drake Development Team, Drake: Model-Based Design and Verification for Robotics,
J. Bongard, V. Zykov, H. Lipson, Resilient machines through continuous self-modeling. Science 314, 1118–1121 (2006).
R. Kwiatkowski, H. Lipson, Task-agnostic self-modeling machines. Sci. Robot. 4, eaau9354 (2019).
A. Sanchez-Gonzalez, N. Heess, J. T. Springenberg, J. Merel, M. Riedmiller, R. Hadsell, P. Battaglia, Graph networks as learnable physics engines for inference and control, in International Conference on Machine Learning (PMLR, 2018), pp. 4470–4479.
B. Chen, Y. Hu, L. Li, S. Cummings, H. Lipson, Smile like you mean it: Driving animatronic robotic face with learned models, in 2021 IEEE International Conference on Robotics and Automation (ICRA) (IEEE, 2021), pp. 2739–2746.
K. Hang, W. G. Bircher, A. S. Morgan, A. M. Dollar, Manipulation for self-identification, and self-identification for better manipulation. Sci. Robot. 6, eabe1321 (2021).
H. E. Gardner, Frames of Mind: The Theory of Multiple Intelligences (Hachette UK, 2011).
J. J. Park, P. Florence, J. Straub, R. Newcombe, S. Lovegrove, Deepsdf: Learning continuous signed distance functions for shape representation, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (IEEE, 2019), pp. 165–174.
V. Sitzmann, J. Martel, A. Bergman, D. Lindell, G. Wetzstein, Implicit neural representa- tions with periodic activation functions. Adv. Neural Inf. Process. Syst. 33, 7462–7473 (2020).
W. E. Lorensen, H. E. Cline, Marching cubes: A high resolution 3D surface construction algorithm. ACM SIGGRAPH Comput. Graph. 21, 163–169 (1987).
S. M. LaValle, Planning Algorithms (Cambridge Univ. Press, 2006).
S. Karaman, E. Frazzoli, Incremental sampling-based algorithms for optimal motion plan- ning. Robot. Sci. Syst. VI, 104 (2010).
M. Tancik, B. Mildenhall, T. Wang, D. Schmidt, P. P. Srinivasan, J. T. Barron, R. Ng, Learned initializations for optimizing coordinate-based neural representations, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2021), pp. 2846–2855.
K. Y. Goldberg, R. Bajcsy, Active touch and robot perception. Cogn. Brain Theory 7, 199–214 (1984).
J. Bongard, H. Lipson, Automatic synthesis of multiple internal models through active exploration, in AAAI Fall Symposium: From Reactive to Anticipatory Cognitive Embodied Systems (AAAI Press, 2005).
J. Bohg, K. Hausman, B. Sankaran, O. Brock, D. Kragic, S. Schaal, G. S. Sukhatme, Interactive perception: Leveraging action in perception and perception in action. IEEE Trans. Robot. 33, 1273–1291 (2017).
S. K. Ramakrishnan, D. Jayaraman, K. Grauman, Emergence of exploratory look-around behaviors through active observation completion. Sci. Rob. 4, eaaw6326 (2019).
J. L. Schönberger, J.-M. Frahm, Structure-from-motion revisited, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2016), pp. 4104–4113.
J. L. Schönberger, E. Zheng, M. Pollefeys, J.-M. Frahm, Pixelwise view selection for unstructured multi-view stereo, in European Conference on Computer Vision (ECCV) (Springer, 2016), pp. 501–518.
D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv:1412.6980 [cs.LG] (22 December 2014).
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, S. Chintala, Pytorch: An imperative style, high-performance deep learning library, in Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, R. Garnett, Eds. (Curran Associates Inc., 2019), pp. 8024–8035.
W. Falcon, PyTorch Lightning Team, PyTorch Lightning,

Information & Authors


Published In

View large Science Robotics cover image
Science Robotics
Volume 7 | Issue 68
July 2022

Submission history

Received: 10 November 2021
Accepted: 17 June 2022


Request permissions for this article.


This work was supported by DARPA MTO Lifelong Learning Machines (L2M) Program W911NF-21-2-0071, NSF NRI Award 1925157, NSF AI Institute for Dynamical Systems 2112085, NSF CAREER Award 2046910, and a gift from Facebook Research.
Author contributions: B.C. and H.L. proposed the research. B.C. developed the main idea, algorithm designs, implementations, simulation, and hardware experiments. H.L. and C.V. provided deep insights and guidance on the algorithm and experiment design. B.C., H.L., and C.V. performed numerical analysis. R.K. provided help on hardware experiments and was involved in the discussions. B.C., H.L., and C.V. wrote the paper. All authors provided feedback.
Competing interests: The authors declare that they have no competing interests.
Data and materials availability: All data and software needed to evaluate the conclusion in the paper are provided at Additional information can be addressed to B.C.



Department of Computer Science, Columbia University, New York, NY, USA.
Roles: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing - original draft, and Writing - review & editing.
Department of Computer Science, Columbia University, New York, NY, USA.
Roles: Conceptualization and Methodology.
Department of Computer Science, Columbia University, New York, NY, USA.
Data Science Institute, Columbia University, New York, NY, USA.
Roles: Conceptualization, Funding acquisition, Methodology, Project administration, Supervision, and Writing - original draft.
Data Science Institute, Columbia University, New York, NY, USA.
Department of Mechanical Engineering, Columbia University, New York, NY, USA.
Roles: Conceptualization, Funding acquisition, Methodology, Project administration, and Writing - review & editing.

Funding Information


Corresponding author. Email: [email protected]

Metrics & Citations


Article Usage


Export citation

Select the format you want to export the citation of this publication.

View Options

Check Access

Log in to view the full text


AAAS login provides access to Science for AAAS Members, and access to other journals in the Science family to users who have purchased individual subscriptions.

Log in via OpenAthens.
Log in via Shibboleth.

More options

Purchase access to this article

Download and print this article within 24 hours for your personal scholarly, research, and educational use.

View options

PDF format

Download this article as a PDF file

Download PDF







Share article link

Share on social media


eLetters is an online forum for ongoing peer review. Submission of eLetters are open to all. eLetters are not edited, proofread, or indexed. Please read our Terms of Service before submitting your own eLetter.

Log In to Submit a Response

No eLetters have been published for this article yet.