Watchmaking: Studying Human Dexterity

We follow a cohort of apprentices in watchmaking over a period of two years. We focus on the acquisition process of dexterous bimanual manipulation skills in watchmaking tasks.

We focus on the assembly and disassembly of three components on the watch face: screw, plate, and spring. Three tools are used during manipulation: screwdriver, tweezers, and stick.

Data collection

We collect both tactile sensing information (applied force) and kinematic information (hand motion) of human subjects during watchmaking tasks. For this purpose, we have established a data collection system, which consists of two systems: Tactile Sensing System and Motion Capture System.

Experimental Setup for Data Collection

For the tactile sensing system, we use three types of sensors to record contact information during manipulation.
(1) Finger TPS Sensors, to record pressure applied on finger phalanges. For each hand, 6 finger TPS sensors are mounted (12 sensors in total) to cover the distal phalanges (thumb, index finger, middle finger) and intermediate phalanges (index finger, middle finger, ring finger).
(2) ATI Nano17 Force/Torque sensor, mounted on the support of the watch face to record the force and torque signals applied to the watch face.
(3) The DigiTacts sensing systems that comprise sensing elements array are mounted on the surfaces of the tool to record applied pressure during manipulation.

The motion capture system is arranged for the recording of kinematic data of human motion: a camera array consisting of 8 GoPro cameras are mounted surrounding the workspace to capture the human hand movement at high speed and high resolution from multiple perspectives. Twenty-five trackers are pasted on each hand to enable the recording of 3D trajectories of each finger joint, which are used to reconstruct the hand skeleton model as well as to reproduce hand motion afterwards.

The video recording work is undertaken by Pomelo Sàrl.

Data analysis and modelling

Breakdown of the Manipulation Time

To analyse the complex human motions during watchmaking task, we segment the recorded human motion data in order to breakdown the entire manipulation process into sequential simple action components. For example, a single complete assembly/disassembly process consists of actions like adjusting base pose, selecting tools, adjusting hand pose, pick and place (of target components), localizing tools, executing assembly/disassembly actions.

Part of the data analysis work is conducted in collaboration with Pomelo Sàrl.

Average Failures per Manipulation

We analyse the collected human motion data from several perspectives. We evaluate and compare the motion of subjects at different skill levels to understand the evolvement of manipulation skills. For example, failures happen during complicated manipulation tasks, such as dropping components, breaking fragile components, failed in localizing tools, and so on. Results from preliminary analysis show that subjects from different skill levels display a significant difference in average failure rate, and failures happen more often in the early stages of learning watchmaking tasks.

Our research also focuses on the failures during executions. We study how human manages to generate new successful motions after the occurrence of failures, in order to develop strategies that enable robots learning from failed demonstrations.

Human Hand Skeleton Model of Manipulating Screwdriver

We reconstruct skeleton models of human hand from the collected kinematic data using OpenSim. Combined with the pressure information recorded on the finger phalanges, we analyse the adjustment of hand pose, synergy of fingers, activities of tendons during different manipulation tasks.
Each human hand has 27 degrees of freedoms (DoFs), some of which are redundant in specified manipulation tasks. We analyse the task constraints to identify the required DoFs for planning the robotic task.
Human changes finger configurations to satisfy different task constraints. As an example, in the manipulation of the screwdriver, three fingers are mainly required: thumb, index finger, and middle finger. The distal phalanx of thumb, along with the distal phalanges of middle and ring fingers, move in collaboration to apply friction to the surface of screwdriver for generating rhythmic rotation movements. In the meantime, the index finger applies constant pressure on the top of the screwdriver to maintain the stability of the screwdriver.

Human Hand Skeleton Model of Manipulating Tweezers for Picking Components

To manipulate tweezers for a pick and place motion, the hand constructs a force closure to realize a stable manipulation as well as to compensate the possible perturbations occur on the tip of tweezers.

Human Hand Skeleton Model of Manipulating Tweezers for Watch Components Disassembly

While using the tweezers for disassembly tasks, subjects often prefer to use fingers to wrap the tweezers, so that the movement of the tweezers along the radial direction is constrained when a force is desired to be applied in the normal direction.

Example: Spring Assembly – A Challenging Bimanual Manipulation Task

The assembly of spring on the watch face is one of the most challenging skills that the apprentices must learn to master during study.

On the one hand, the spring is a very tiny component on the watch face, thus motion of high precision is required during the pick and place movement. On the other hand, the spring must be properly deformed during assembly, in order to save elastic potential energy in it. Therefore, appropriate force must be applied (usually by the dominant hand with tweezers) in the correct direction, while the position of the spring must be maintained (usually by the assistant hand, using the wooden stick) unchanged in the meantime.

The correct way to assemble the spring is demonstrated in the following video.

Either inappropriate manipulation direction or improper application of force may result in failures, even the damage of components. Following videos illustrate the typical failures occur during the assembly: the spring flies away.

Hand posture in bimanual fine manipulation tasks

Many manipulations in human everyday activities rely on the collaboration of our two hands. Bimanual fine manipulation tasks are particularly demanding as precise coordination among fingers of both hands is required. Yet, despite the abundant degrees of freedom (DoFs) need
to be controlled, humans can perform such tasks nearly effortlessly. Ample evidence suggests that the human central nervous system (CNS) predetermines hand poses that are compatible with task conditions (e.g. task objective, physical properties of objects, control of force/torque
required by task) prior to manipulation. However, the mechanism by which the CNS encodes task conditions and translates such information into appropriate motor commands is poorly understood.

We designed a comparative experiment to infer task conditions’ effect on human hand control strategies. Twenty right-handed subjects (aged 24.2+/-6.0, 5 women) were tasked to a precise unscrewing task consisting of two steps: placing a screwdriver on a tiny (1.6mm) screw that is tightly mounted on a watch face, and unscrewing it without sliding off. Subjects were equally assigned to two groups and performed the same task in two conditions. In the free-base condition, the watch face was let free to move on the table; whereas in the fixed-base condition, it was fixed on the table. 

Subjects employed similar hand poses across the two successive task steps (placing and unscrewing) within the same conditions, whereas distinct hand poses were observed between these two conditions. This suggests a generic task-oriented motion planning prior to execution.
In the free-base condition, subjects consistently chose similar hand poses across trials. They used each hand to manipulate one piece of equipment, as required to control for more DoFs. In contrast, subjects used both hands to control the screwdriver in the fixed-base condition, and showed diversity in hand pose combinations. Analysis indicates that subjects select hand poses to control simultaneously the task-relevant DoFs (i.e. demanded by the task)
and the task-irrelevant DoFs. The latter, although not crucial to the task goal, may bring in perturbations and impair performance if not properly controlled. Moreover, subjects manifested a tendency to balance virtual fingers (fingers that move together as a functional unity) across the two hands, such that the control of task-demanded force and torque is decoupled. This strategy may be designed to achieve better task performance while reducing the control complexity.

Distribution of forces in bimanual assembly task

Skills that require the collaboration of both arms are common in humans’ everyday activities, and the coordination of upper limbs is key to human bimanual dexterity. In this study, we focus on the task of assembling a watch spring, where precise, coordinated, and directionally compliant control of the hands and fingers motions is required. We recorded the motion of subjects and the force/torque applied on the watch face during manipulation tasks.

We compare the performance of 15 subjects participating in our experiments (10 novices and 5 experts). Our analysis shows that experts try to align the direction of force application demanded by the task with the optimal force application direction of the dominant arm. In addition, experts adjust finger placements to improve the dexterity of hands for manipulation. We assume the differences in performance across novices and experts can be explained by the differences in the weights of criteria (manipulability, task compatibility, and discomfort), which are used to evaluate the motion. We develop a computational model to infer the weights. Preliminary results show that novice subjects put higher weight on the task compatibility, while experts have higher weight on motion comfort.

Publications