From safe motion to publishable evidence.

Record the discovered graph

Identify the R1 joint-state, IMU, battery and remote-controller streams from the inventory. Public SDK2 examples show these state classes exist, but their R1 topic names are TO-VERIFY.

Echo and bag only verified streams

ros2 topic echo <verified_joint_state_topic> --once
ros2 bag record \
  <verified_joint_state_topic> \
  <verified_imu_topic> \
  <verified_battery_topic> \
  <verified_remote_topic>

Map all 24 controlled joints

Confirm 6 per leg, 2 waist and 5 per arm; record index, name, sign, units, neutral angle and delivered limits. If head joints appear, add them separately rather than altering the confirmed 24-DoF baseline.

Mechanical gate

Mount the gantry, clear everyone beyond full arm reach, keep the remote in the spotter’s hands and start above the session’s minimum battery threshold.

Control-ownership gate

Use the vendor/InDro procedure to release only the target arm group. Unitree’s public SDK2 Python guidance explicitly says high-level sport_mode must be disabled before low-level motor control. Verify the R1 equivalent on the unit.

Command envelope

Start from measured position, move one shoulder joint only a few degrees with conservative gains and velocity, continuously publish the required hold/trajectory stream, and reject stale/NaN/out-of-range commands.

Stop and inspect

Return to the captured neutral pose; stop publishing through the documented transition; inspect error, current/torque estimate and temperature. Repeat only after a clean log.

Author in named joint space

Use only the five verified right-arm joints. Every waypoint includes time-from-start; interpolate smoothly, cap position/velocity and begin/end at NEUTRAL_V1.

Run progressively

Offline plot → command shadow log → gantry + one joint → gantry + arm → ten full repetitions. Keep legs under the vendor balance controller; never mix a custom arm loop with unverified whole-body commands.

Version the behavior

Store joint map, radians, timing, interpolation, limit file, controller mode and firmware/SDK version with the trajectory.

Camera quick test

The official SDK2 Python repository supplies example/front_camera/camera_opencv.py; it is not documented there as R1-specific. Run read-only after the network interface is known.

python3 ./example/front_camera/camera_opencv.py <robot_network_interface>

XR image path

xr_teleoperate uses teleimager for physical-robot image service and supports UVC/V4L2/GStreamer/RealSense sources. Configure the actual binocular devices; confirm left/right ordering, resolution, FPS and timestamp source.

Audio route

On the InDro image, enumerate capture/playback devices and ROS audio nodes. Verify sample rate, channel mapping, echo cancellation and speaker latency. The generic SDK example controls volume, but a public R1 mic-stream API was not verified: label it TO-VERIFY.

Before any transition

Robot on gantry; arms near initial pose; spotter has remote; no participant present; control process heartbeats visible; current mode read back from the robot.

XR motion mode

Unitree’s R1-capable XR project supports regular mode only; running mode is explicitly unsupported. In motion mode, both XR joysticks pressed invoke a soft emergency stop to damping mode, while the handheld R3 controller remains required for robot control mode.

Exit cleanly

Bring arms near initial pose before quitting. The official XR path returns arms over five seconds before ending control. Confirm the mode change on telemetry; do not assume process exit means motors are safe.

Test failure paths

With gantry and empty zone, test network loss, stale command, policy crash, operator exit, soft stop and physical power isolation. Write the observed robot response and restart sequence.

Inventory versions

Record host OS, R1 firmware, SDK/ROS package versions and commits, network interface, robot address/topic namespace, camera resolution and nominal update rates. Do not upgrade during a study.

Read joints

With motion disabled, stream timestamped joint positions, velocities and controller state for 60 seconds. Confirm count, order, units and signs against the delivered R1 documentation.

Read vision and audio

Open one camera stream and the microphone path. Check image orientation, frame age and whether audio capture conflicts with the voice pipeline.

Measure clock drift

Compare robot, logging desktop and sensor-host clocks at start and end of the 60-second run. Pick one master clock and log offsets.

Save neutral

Capture the approved arm, torso and head pose. Every gesture begins and ends here. Record joint-space tolerance for “neutral reached.”

Author one joint trajectory

Begin with a small right-arm wave. Use waypoints and eased interpolation. Keep locomotion disabled and body balance logic outside the gesture controller.

{
  "id": "WAVE_SMALL_V1",
  "start": "NEUTRAL_V1",
  "max_duration_s": 4,
  "allowed_groups": ["right_arm"],
  "on_timeout": "RETURN_NEUTRAL"
}

Validate progressively

Log-only → one joint at low gain → one arm → full gesture. Run ten repetitions; inspect peak position, peak velocity, return error and any balance response.

Build the bounded catalog

Add nod, bow, point-left/right, beckon, present and hesitation only after the wave passes. Each behavior gets version, start/end pose, duration, degrees of freedom, stop rule and video reference.

Define the contract

Ollama returns speech text, one allow-listed gesture ID and a timing relation. Reject malformed output or unknown IDs to NO_ACTION.

{
  "say": "Welcome to the RAP Lab.",
  "gesture": "WAVE_SMALL_V1",
  "timing": "gesture_then_speak"
}

Test disconnected

Run at least 100 benign, paraphrased, ambiguous and adversarial utterances. Score correct skill, wrong skill and safe rejection. The robot stays offline.

Run shadow mode

Connect state streams but send no commands. Log selected behavior, safety checks and intended timing. Human reviewers approve the first 20 correct selections.

Enable supervised execution

Orchestrator order: stop listening → validate current pose and clearance → reserve controller → run gesture → play speech according to condition → return neutral → release controller → resume listening.

Verify the interface and input device

Confirm delivered arm configuration before choosing any R1 arm profile. Confirm whether the lab has a compatible XR headset; that remains an open inventory question. Without one, record scripted demonstrations now and add XR later.

Define episode schema

For each frame store monotonic timestamp, camera image, arm joint position/velocity, commanded action, controller state, behavior label and operator event. Save session metadata once, not redundantly in every frame.

Standardize resets

Mark floor position and camera tripod; run RETURN_NEUTRAL; confirm clearance; start recording; execute exactly two wave cycles; stop; annotate pass/failure; reset.

Collect in blocks

Collect ten episodes, inspect them, then continue toward fifty clean demonstrations. Do not wait until episode fifty to discover swapped joints or missing frames.

Implement the observation contract

Expose fixed keys for camera image and selected arm state. Normalize only with statistics saved from the training split. Keep units and joint order explicit.

observation = {
  "observation.images.head": rgb_frame,
  "observation.state": ordered_arm_joint_positions
}

Implement the action firewall

Check shape and freshness; clip to tested joint range; rate-limit velocity; reject NaN/Inf; enforce timeout; verify controller ownership; return neutral on fault. The neural policy never bypasses this layer.

Convert and inspect

Split by start condition, not random frame. Plot every state/action channel, review each video and retain a separate held-out condition. Generate dataset statistics only from training episodes.

Train the simplest baseline

Use an ACT imitation policy first. Save config, code commit, seed, data version, checkpoint and training curves. Loss is diagnostic, not the outcome.

Deploy progressively

Offline inference → shadow actions → one joint → one arm → full gesture. Compare scripted and learned variants on familiar and held-out starts.

Choose sensors by question

Use the ring alone for lower-burden HR/HRV + EDA. Add OpenBCI when the theory requires EEG. In 16-channel mode Cyton + Daisy samples at 125 Hz, suitable for band-power and limited ERP-style work, not high-frequency claims.

Prepare participant and baseline

Check consent and contraindications; fit ring and electrodes; inspect contact and signal quality; seat participant; record a quiet baseline before any robot movement.

Emit event markers

Use one event dictionary across robot, task UI and acquisition host: session start, baseline start/end, robot speech start/end, gesture start/end, advice, error, repair, participant response and trial end.

TRIAL_017 | t_master | CONDITION=APOLOGY_BOW
ROBOT_ERROR_ONSET
REPAIR_SPEECH_ONSET
REPAIR_GESTURE_ONSET
PARTICIPANT_CHOICE
TRIAL_END

Run a synchronization bench test

Send repeated markers while recording all streams; estimate offset, jitter and missing markers. The ring’s iOS/cloud path may be the bottleneck—verify local raw export before committing to event-level analyses.

Engineering pilot

Team members only. Stress start/stop, lost network, stale camera, repeated gestures, low battery, voice timeout and sensor reconnect. No human-subject measures.

Manipulation check

Test whether intended constructs differ—such as confident vs hesitant or apology vs no repair—without unintended differences in loudness, duration, danger or perceived malfunction.

Protocol pilot

Run consent through debrief, including randomization, sensors, questionnaire, interview and file handoff. Time every segment and note deviations.

Version stimuli

Export robot trajectories, speech audio/text, prompts, UI, questionnaires and randomization table. Compute hashes and record the deployed versions.

Preregister decisions

State hypotheses, inclusion/exclusion, manipulation checks, primary DVs, processing choices, artifact rules, stopping rule and analysis model before collecting confirmatory data.

Rehearse staff

Each operator completes the same certification run: setup, baseline, condition delivery, abort, recovery, notes and teardown.

Lock change control

No software, firmware, prompt, timing or questionnaire changes mid-study without documented version break and analysis decision.