Learning Robot Fine and Dexterous Manipulation: Perception and Control

  • When: November 9th, 2024
  • Where: CoRL 2024, Science Congress Center Munich, Walther-von-Dyck Str. 10
  • Format: Full-day workshop
  • Organized by:
    • Soheil Gholami, Ph.D.
      • Learning Algorithms and Systems Laboratory, EPFL
    • Xiao Gao, Ph.D.
      • Learning Algorithms and Systems Laboratory, EPFL
    • Kunpeng Yao, Ph.D.
      • Newman Laboratory, MIT
    • Aude Billard, Full Professor
      • Learning Algorithms and Systems Laboratory, EPFL

Outline

Objectives and Scope

Fine manipulation involves making precise movements with robotic hands and fingers to handle small objects or perform intricate tasks, such as threading a needle. Dexterous manipulation goes further, requiring highly skilled, accurate, and versatile manipulation of objects through complex interactions among multiple fingers and joints. The ability to achieve fine and dexterous manipulation with high speed, accuracy, and dexterity is becoming increasingly important in robotics research. However, it also poses many challenges, such as frequent making and breaking of contact, real-time feedback control with high-dimensional observations, high-dimensional control spaces, and objects being in unstable configurations. Traditional methods rely on precise robot and environment models but often struggle with real-world uncertainties and lack generalizability. Despite decades of research, most demonstrations of dexterous manipulation still rely heavily on teleoperation. Achieving robust and generalizable dexterous manipulation requires advancements in perception integration, data collection, and control. Advances in robot learning, including machine learning and transfer learning, offer promising pathways to enhance robotic performance in fine and dexterous manipulation tasks. This event seeks to convene researchers from diverse disciplines to share insights on pushing this critical boundary.

This workshop aims to bring together junior and senior researchers to discuss the latest advancements, challenges, and future directions in learning-based approaches for robot fine manipulation skills, one of the most challenging areas in robotics. We will delve into the current state-of-the-art across relevant areas, including the hardware and mechanical design of dexterous manipulators, generalizable skill learning techniques, and sensing modalities such as tactile sensors and vision systems. Researchers will have opportunities to present posters, give contributed talks, and engage in thought-provoking discussions.

We will explore the following focused research questions:

Visual Perception

  • In the context of dexterous manipulation, how can we overcome challenges related to occlusions between objects and robot hands?
  • What are the strategies for generalizing manipulation policies to outdoor or unstructured environments, considering variables such as lighting changes and the need to process extensive information?

Tactile Perception

  • How can tactile feedback be utilized in dexterous manipulation?
  • What types of tactile sensors and data are most beneficial for these tasks, and how can they complement or substitute visual feedback?

Robot Skill Learning

  • Foundation vs. Specialized Models: Is the future of dexterous manipulation in developing a foundation model for general tasks, or will specialized models for specific tasks prevail?
  • Dynamic Tasks: How can learning-based approaches effectively manage dynamic manipulation that necessitates precise control and accurate dynamics modeling?
  • Generalization of Human Hand Data: Can we standardize the collection of human hand data for dexterous manipulation using expert-grade equipment, and what strategies can bridge the data gap between human and robotic manipulation capabilities?
  • Enhancing Data Collection Methods: What improvements can be made to teleoperation and other data collection methods to enable large-scale acquisition of manipulation data?
  • Reducing Reward Engineering for Reinforcement Learning: Tuning reward functions is non-trivial for new tasks. How can we reduce the engineering effort in reward shaping? How can we scale up reinforcement learning pipelines to solve hundreds of different tasks?

Learning Fine and Dexterous Manipulation Skills

  • What are the main challenges in learning fine and dexterous manipulation skills compared to other robot skills?
  • Is end-to-end learning effective for acquiring these skills?
  • What challenges arise in coordinating surplus degrees of freedom during in-hand manipulation, and what are the requirements for learning techniques to address these challenges effectively?
  • How can various applications, such as prosthetics, benefit from emerging algorithms?
  • What challenges and prerequisites do touch processing and tactile sensing entail?
  • How can we establish robust benchmarks and frameworks for systematically evaluating and comparing the effectiveness of various learning techniques?

The expected attendees for these discussions and talks are researchers from academia and industry specializing in robotics and machine learning, particularly those interested in applying learning approaches to achieve fine and dexterous manipulation in robots.

Call for Contribution

We invite you to submit 4-page, single-column abstracts (including figures, tables, and references). Please provide the required information and upload your PDF abstract using the form below.

We recommend submitting a 4-page abstract in PDF format, following the CoRL template (one column). All submissions will undergo a peer-review process and will be evaluated based on their relevance and contribution to the workshop’s topics. Accepted submissions will be featured at the workshop through Poster Spotlight Talks (2-minute presentations) and Poster Sessions. They will also be published on the workshop’s websites, along with an optional 2-minute video summary.

Keywords specific to the event

  • In-hand manipulation, dexterous manipulation, fine manipulation
  • Learning robot fine-manipulation skills
  • Reinforcement learning in robotic fine-manipulation
  • Transfer learning in robotic fine-manipulation
  • Imitation learning/Learning from demonstration of fine-manipulation skills
  • Data collection for fine and/or dexterous manipulation
  • Simulation for fine and/or dexterous manipulation (sim2real approaches)
  • Investigating planning strategies and algorithms for dexterous manipulation tasks
  • Integrating tactile sensing for enhanced manipulation and multi-modal learning

Important dates

Speakers and Agenda (To be updated soon)

List of Speakers (in-person presentations):

(The schedule will be updated soon!)

Time slot Talk Comment
8:30 to 8:40 Organizers: Welcome and Introduction  

Acknowledgment

This workshop is supported by the euROBIN project (grant agreement No 101070596).

(This workshop is the result of combining two similar workshop proposals that were both accepted to be held at CoRL 2024. Link to the other workshop webpage)