Learning Contact Representations from Zero-Cost Proprioceptive Effort Signals for Contact-Rich Visuomotor Imitation Learning

Jaydeepkumar Ranpariya · Master's thesis · FAU Erlangen-Nürnberg and Fraunhofer IIS · 2026

Abstract

Contact-rich manipulation depends not only on where a robot moves, but also on how it interacts with its surroundings. Near contact, the gripper or object can obscure the relevant region from the camera, while joint positions alone cannot reveal whether an object has been secured or a peg is pressing against the rim of a hole. Force/torque and tactile sensors can provide this missing information, but they add hardware and integration effort. Low-cost bus servos already expose motor-current readings without additional sensing hardware; however, these uncalibrated signals combine external contact with pose-dependent loading, motion, friction, and transmission effects. It therefore remains unclear whether they can provide a useful additional observation for contact-rich visuomotor imitation learning.

This thesis integrates the six current registers of an SO-101 robot into a controlled imitation-learning pipeline. The signals are first characterised under static, known-load, scripted-contact, natural-motion, and insertion conditions. A synchronised dataset of 100 vertical peg-in-hole demonstrations is then collected through leader–follower teleoperation, with wrist images, joint positions, action targets, and motor current recorded on a common timebase. The same demonstrations are used to compare vision-and-position policies with current-aware variants. Within the Action Chunking Transformer family, instantaneous concatenation is compared with a learned joint-selective temporal representation; a Diffusion Policy comparison examines current conditioning under a different action-generation architecture.

Counterfactual input interventions test whether trained policies respond to current, while repeated physical trials, failure-phase analysis, and session comparisons evaluate closed-loop behaviour. By linking signal characterisation, synchronised data collection, policy representation, modality-use diagnostics, and physical evaluation, the thesis provides a systematic framework for assessing weak proprioceptive effort signals as policy inputs. It contributes a multimodal peg-in-hole dataset and software framework, platform-specific evidence about the ST3215 current registers, and controlled methods for distinguishing information in an additional modality from model dependence and behavioural benefit.

1 Introduction

Contact-rich manipulation is harder because the decisive state may be physical rather than purely geometric. A gripper can close without securing an object, a peg can reach a hole while pressing against its rim, and two nearly identical camera images can precede either a smooth insertion or a jam. In each case, the robot must respond to an interaction that is only partly visible.

The SO-101 used in this study is fitted with Waveshare ST3215 servos, whose control table specifies a read-only motor-current register. Reading that existing register adds no sensor or mechanical modification to the study arm. Its interpretation is nevertheless difficult because motor-side current is affected by pose, motion, friction, and the high-ratio transmission, and it does not directly specify the direction or location of an external force.

Research question

Can weak, uncalibrated motor-current signals already available on a low-cost robot provide a useful learned contact representation for contact-rich visuomotor imitation learning?

Three questions make the investigation operational.

• Does motor current contain task-relevant information? This requires the registers to be available at the policy rate and to exhibit repeatable variation under contact-relevant conditions beyond their quiet-state variation.
• Do trained policies respond to current? This asks whether changing the current input while holding image and position fixed changes the actions predicted by a trained policy.
• Does adding current improve closed-loop task performance? This compares policies with and without current during repeated physical peg-in-hole trials and examines where failures occur.

Contributions

• a reproducible characterisation of all six ST3215 current channels as candidate learning inputs, covering acquisition, distribution, pose dependence, contact discrimination, and temporal behaviour;
• an extended LeRobot-compatible schema and a synchronised 100-episode multimodal peg-in-hole dataset containing camera observations, follower joint positions, leader-derived actions, and follower current readings, together with human-reviewed phase annotations for the analysed subset;
• controlled ACT implementations for direct current concatenation, training-derived current preprocessing, and a learned joint-selective temporal representation, supplemented by clearly separated auxiliary conditioning methods;
• a Diffusion Policy vision-and-position system and a learned-current counterpart evaluated under a shared dataset and execution timebase;
• a model-agnostic counterfactual diagnostic that measures how predicted actions change when one observation modality is replaced while the remaining inputs are held fixed; and
• a software and verification framework that reproduces the numerical analyses, plots, observation transformations, and evaluated policy settings.

Scope

The investigation covers one low-cost manipulator, one primary peg-in-hole task, one demonstration dataset, and the evaluated ACT and Diffusion Policy implementations. Motor current is treated as a policy observation and is not converted into an estimate of Cartesian force or joint torque. The robot is not augmented with tactile, acoustic, inertial, or force/torque sensors, and no simulated current model or sim-to-real transfer is introduced.

2 Related work

This thesis lies between two established research directions: estimating physical interaction from signals inside a robot, and incorporating interaction measurements into learned manipulation policies. The first direction determines what motor current can reveal about contact and loading. The second determines how force, torque, tactile, or effort observations can influence action prediction.

Research gap

Prior work has established three relevant capabilities. Motor-side signals can support force or collision estimation when combined with models, calibration, or force-labelled supervision. Force, torque, and tactile observations can improve learned contact-rich manipulation through direct fusion, structured tokens, conditional modulation, and corrective control. Contact-aware sampling, gating, and curricula can strengthen the influence of an interaction branch during learning.

This leaves low-cost robot learning in an unresolved position: the platform may already expose a contact-related signal, but its usefulness to a learned controller has not been established without dynamics, calibration, or force-labelled supervision.

3 Background

Learning a manipulation policy from demonstrations requires a model of both the physical interaction and the information available to the robot. The robot observes that interaction through imperfect measurements, predicts a sequence of actions, and may reach situations that were not represented in the demonstrations.

Modality utilisation and collapse

Adding a modality to the input does not guarantee that the trained policy uses it. Multimodal networks can favour a representation that learns faster or already explains much of the target, leaving another modality weakly represented even when it contains complementary information.

Evaluating an added policy modality involves three distinct levels: whether the current signal contains task-relevant information in the recorded data, whether a trained policy changes its predicted actions when that input changes, and whether including the signal improves closed-loop task performance. These levels require different evidence: signal characterisation for recorded information, counterfactual input tests for model dependence, and matched real-robot comparisons for behavioural benefit.

4 Effort-signal characterisation

Before motor current can be added to a policy, it must first be shown to carry a repeatable and task-relevant signal. The analysis considers whether the register can be sampled at the policy rate, how its values are distributed, how strongly they depend on pose, how noisy they are, and whether their temporal behaviour distinguishes contact-relevant conditions.

On this arm and task, motor current is suitable for evaluation as an additional policy input because the ST3215 register distinguishes contact-relevant conditions. At the same time, the signal is pose-dependent, zero-inflated in the training data, integer-quantised, and neither a calibrated force measurement nor a directional contact measurement.

5 Multimodal dataset

The final task is cylindrical peg-in-hole insertion with a peg diameter of 16.45 mm and a bore diameter of 16.8 mm, both measured directly with calipers during setup. This gives a diametral clearance of 0.35 mm, or c/d = 0.021, at an entry without a chamfer.

Demonstrations were recorded with the LeRobot leader-follower interface and two mechanically similar SO-101 arms. The author moved the passive leader by hand while the follower tracked the six leader joint targets at a configured rate of 30 Hz. The primary dataset combines two 50-episode vertical-insertion batches recorded with the same task and schema. Each source batch contains two collection sessions of 25 episodes, giving four sessions in total.

Current distribution and capping

Across the six channels, 44.9% of the values are zero, and the per-joint zero fractions range from 34.6 to 54.0%. The shoulder-pan channel has a median of zero and a 99.5th-percentile value of 4 counts, but reaches 52 counts. The selected representation caps each joint independently at the 99.5th percentile of its training distribution. The fitted caps are 4, 33, 32, 8, 3, and 33 counts for shoulder pan, shoulder lift, elbow flexion, wrist flexion, wrist roll, and gripper, respectively.

Phase annotations are stored separately and are never policy inputs. The annotation table contains 32 episodes reviewed against video. One row records its phase boundaries out of order and is excluded, leaving 31 episodes.

6 Multimodal policy learning method

The first comparison adds the current measured at the present time step to an otherwise unchanged Action Chunking Transformer. A second ACT variant replaces this direct concatenation with learned current histories for the individual joints, allowing the policy to capture short temporal patterns. The final comparison adds a learned current route to Diffusion Policy and examines the same modality under a different approach to action generation.

Action Chunking Transformer

The 640×480 wrist image passes through a ResNet-18 backbone. Its final 512×15×20 feature map is projected to 300 visual tokens of width 256. The six joint positions are linearly projected to one state token of the same width. A four-layer, eight-head Transformer encoder combines these tokens with a 32-dimensional latent token. A one-layer Transformer decoder uses 50 learned queries to predict an absolute action chunk. The loss is the padding-masked L1 action error plus the Kullback-Leibler term with weight 10.

Diffusion Policy

Diffusion Policy generates an action trajectory by iteratively denoising a noisy sequence conditioned on the robot observation. Its one-dimensional conditional U-Net uses widths (512, 1024, 2048), kernel width 5, GroupNorm with eight groups, skip connections, and a 128-dimensional diffusion-step embedding. The evaluated DP-V receives two observation times separated by 0.1 s. Each wrist image is resized to 144×192 without cropping and encoded by a ResNet-18 with GroupNorm.

Modality-use diagnostic

A multimodal input can be present without affecting the policy. To test whether the trained model responds to current, the same preprocessed observation is evaluated twice. The first pass uses the recorded current. The second replaces every instantaneous and temporal current coordinate with the corresponding training-set mean in the checkpoint's normalised space. Positions and images remain unchanged.

7 Evaluation

Protocol

All principal comparisons use the same SO-101 follower arm, peg, insertion fixture, and 100-demonstration dataset. The peg begins at one marked position for both policies. One operator resets and scores every trial. A 30-trial comparison contains five blocks, each with six trials per policy. The first policy alternates between blocks in both normalised ACT sessions, the joint-selective temporal ACT session, and the final Diffusion Policy session. Only the raw ACT comparison and the explicitly exploratory screens use a fixed order. The timeout is 45 s.

Success proportions are reported with Wilson 95% confidence intervals. Two policies evaluated within one session are compared with a two-sided Fisher exact test, and differences in success proportions use Newcombe–Wilson 95% confidence intervals. An absolute difference of 10 percentage points was fixed before the normalised comparisons as the smallest change considered practically relevant.

Task selection

The task was refined sequentially until the vision-and-position baseline left meaningful room for an insertion-stage improvement. The rows therefore identify the selected operating point rather than estimate isolated effects of those factors.

Modality-use analysis

The normalised direct ACT-M checkpoint gives a mean relative action change of 0.0795 over 510 phase-stratified demonstration frames. The residual-conditioned DP-M checkpoint gives 0.00649 over 150 frames. For context, replacing joint position by its training mean and setting the image to zero produce relative action changes of 0.403 and 0.445 for ACT-M-norm; the corresponding values for DP-M are 0.681 and 0.113. The diagnostic establishes that current reaches the action output in the two retained principal checkpoints for which the test is available. It does not show whether the induced action change is useful.

Principal closed-loop comparisons

For interpretation, the relative change in success is 100(p_M/p_V − 1), where p_M and p_V are the observed current-aware and contemporaneous vision-and-position success proportions. Relative changes describe the size and direction of the observed result; the absolute rates, counts, and difference intervals retain its scale and uncertainty.

The current-aware setting records a 71.4% relative increase in success over its contemporaneous baseline, corresponding to five additional successes and an absolute gain of 16.7 percentage points. This is the largest favourable relative and absolute point estimate among the four principal comparisons, and its absolute magnitude exceeds the ten-point relevance criterion introduced for the normalised comparison. The Newcombe–Wilson 95% interval spans −6.7 to +37.8 percentage points and Fisher's exact test gives p = 0.267. The observed result favours the current-aware policy, while the interval leaves superiority unresolved. The current-aware policy scores higher in three blocks and ties its control in two; these blocks describe the same checkpoint pair within one logical evaluation session and are not independent replications.

Relative to the contemporaneous ACT-V success rate, the raw-current system records 57.9% lower success. This is the only principal interval that excludes zero, and it points towards harm. It is also the only principal session in which ACT-V always ran before ACT-M within a block. The observed Diffusion Policy result favours the current-aware system by 33.3% relative to its contemporaneous baseline, but the uncertainty leaves superiority unresolved, and the two designs have different trainable parameter sets. As a retrospective sensitivity check across the four principal pairwise tests, Holm correction leaves the raw ACT difference below 0.05 (adjusted p = 0.0355); the other comparisons remain unresolved.

Normalised direct ACT in two interleaved sessions

The first session is an 11.8% relative decrease, or −6.7 percentage points with a Newcombe–Wilson 95% interval of [−29.9, 17.6] points and Fisher's exact p = 0.796. This completed 30-trial-per-policy session supplies the principal normalised ACT comparison. The second session produces 10/20 and 6/20. It is retained as a separate robustness check, with the same negative direction and Fisher's exact p = 0.333. The two sessions retain a negative observed success difference after normalisation, without establishing the magnitude of a disadvantage or a reproducible retention effect.

Failure phases

No pattern repeats across policy families strongly enough to support a phase-specific benefit from current.

Session variation

The unchanged ACT-V checkpoint scored 17/30 in the first normalised direct session, 7/30 beside joint-selective ACT-M, and 10/20 in the second normalised direct session. In the final Diffusion Policy session, combined success falls monotonically from 8/12 in the first block to 6/12, 4/12, 2/12, and 1/12. These patterns show why a contemporaneous control is essential and why absolute success counts from separate days cannot be used to rank current representations.

8 Discussion

Existing motor-current readings on the evaluated SO-101 contain contact-related information and can influence learned action predictions. The strongest favourable task result comes from joint-selective temporal ACT: success increases from 23.3% to 40.0%, a 71.4% relative improvement over its contemporaneous vision-and-position control. This observation supports further investigation of the current-aware setting, while its uncertainty leaves superiority unresolved.

Interpretation of the main findings

The temporal ACT comparison provides the clearest favourable task observation. Its 71.4% relative increase corresponds to 12/30 successes rather than 7/30, or five additional successes and an absolute gain of 16.7 percentage points. The temporal gain was recorded in the session with the lowest success rate for the unchanged ACT-V checkpoint, so the relative magnitude must be read against that session's baseline. Residual-conditioned Diffusion Policy also has a favourable point estimate, with a 33.3% relative increase, whereas direct concatenation shows no observed benefit in either principal ACT comparison. The study separates signal informativeness, policy sensitivity and observed task benefit without establishing that learned fusion is superior to direct concatenation.

Representation, demonstrations and execution

Several explanations are consistent with the observed outcomes, and the retained evidence does not select a single mechanism. One possibility is competition between representations during behavioural cloning: images and joint positions are available throughout the demonstrations, whereas current is unevenly informative across the trajectory. A second possibility is that current reveals loading without specifying the required correction. Temporal ACT drops the peg less often than its control, but loses more trials before insertion; this motivates a retention or action-selection hypothesis without establishing that current caused either change. A third possibility concerns the opportunity to use newly observed contact: for temporal ACT, the 0.30 s current history is shorter than the 1.67 s execution interval, so a brief event can leave the history before the next prediction. The demonstration process supplies a fourth possible constraint, since the operator controlled the leader arm visually and did not receive the follower's current as feedback.

Limitations and unanswered questions

Each trained condition uses one random seed, the demonstrations and evaluations involve one operator, and each principal comparison contains 30 trials per policy. The resulting intervals leave substantial uncertainty about modest benefits, and no independent replication of the favourable temporal ACT comparison was completed. The raw direct ACT session uses fixed within-block order, limiting attribution of its adverse difference to current. The completed Diffusion Policy systems differ in both current conditioning and trainable parameters. The exact principal temporal-ACT weights and resolved training command were not retained locally, preventing complete reproduction and checkpoint-specific mechanism analysis. The findings apply to one task, one SO-101, one servo family and one demonstration dataset. Motor current also leaves an observability gap for local contact and slip.

9 Conclusion and future work

The joint-selective temporal ACT comparison favours the current-aware setting: it recorded 12/30 successes (40.0%) against 7/30 (23.3%) for its contemporaneous ACT-V control. This is a 71.4% relative increase, corresponding to five additional successes and an absolute gain of 16.7 percentage points. The observed magnitude exceeds the ten-point relevance criterion, but its 95% difference interval of [−6.7, +37.8] points leaves superiority unresolved, and no independent replication was completed.

Direct fusion has a different observed direction. Raw direct-concatenation ACT recorded 8/30 successes against 19/30 for ACT-V, a 57.9% relative decrease in a fixed-order session whose full difference cannot be attributed to current alone. The principal normalised comparison recorded 15/30 against 17/30, an 11.8% relative decrease; a separately reported robustness check retained that negative direction. The selected current-aware Diffusion Policy system achieved 12/30 successes and the visual comparator achieved 9/30, a 33.3% relative increase, with a difference interval that includes both benefit and harm.

Signal informativeness is demonstrated on this platform, and policy sensitivity is demonstrated for the checkpoints to which the counterfactual could be applied. The learned temporal ACT and residual-conditioned Diffusion Policy settings have favourable observed success changes, with the largest relative gain in temporal ACT, while reliable superiority and the causal contribution of current remain unresolved. Together, these contributions establish the information content of zero-cost current and identify a promising learned temporal setting whose possible performance benefit requires more precise, independently replicated evaluation.

• The most direct extension is to repeat the principal comparisons with multiple training seeds, tasks, sessions, and robot instances.
• Higher-rate or asynchronous current histories could preserve brief events, while shorter execution horizons could give newly observed contact an earlier influence on the next action plan.
• Temporal encoders may be pretrained on independently labelled contact events, and phase-aware or uncertainty-aware fusion could regulate when effort information enters the policy.
• Future datasets should include repeated, labelled examples of contact-conditioned correction and recovery.