craic/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

CRAIC (Camera-Robot AI Control System) — Competition code for the China Robot and Artificial Intelligence Competition (中国机器人及人工智能大赛), Robot Task Challenge (Small Desktop Level).

Hardware: ESP32-S3-WROOM-1-N16R8 with OV2640 camera, 6-DOF mechanical arm with Feettech SCS/STS serial servos.

Architecture: Three-tier system communicating via UDP port 8888:

1. ESP32-S3 firmware (jxbeye/) — Dual-core camera streaming + UDP command receiver
2. ROS 2 teleop (ros2/src/udp_teleop/) — Keyboard control node for chassis and arm
3. Python tools (tools/) — Standalone control scripts with inverse kinematics

Build & Run Commands

ESP32-S3 Firmware

cd jxbeye
pio run -t upload          # Build and flash firmware
pio device monitor         # Serial monitor (1000000 baud)

First boot: ESP32 creates AP ESP32-S3-Camera (password 12345678), access web UI at http://192.168.4.1.

WiFi configuration via serial: Send WIFI:SSID:PASSWORD to configure station mode.

ROS 2 Teleop

# Build (from ros2/ directory)
conda activate ros2_humble
colcon build --symlink-install --packages-select udp_teleop
source install/setup.bash

# Run keyboard control
ros2 run udp_teleop keyboard_control \
    --ros-args --params-file src/udp_teleop/config/params.yaml

# Override target IP
ros2 run udp_teleop keyboard_control \
    --ros-args -p udp_ip:=192.168.4.1 -p udp_port:=8888

Keyboard mappings:

• Chassis: W/S (forward/back), A/D (strafe), Q/E (rotate)
• Arm: ↑/↓ (height), 2-6 (select joint), ←/→ (adjust angle)

Important: Must use ros2 run, not ros2 launch — the stdin keyboard backend requires an interactive terminal.

Python Arm Control Tools

# Direct joint command with interpolation
python tools/udp_control.py joints \
    --height -100 --j2 10 --j3 20 --j4 30 \
    --duration 1.0 --rate 20

# Cartesian pose mode (uses inverse kinematics)
python tools/udp_control.py pose \
    --x 150 --y 50 --z -100 --phi 45 \
    --duration 1.0

# Dry run (print commands without sending)
python tools/udp_control.py pose --x 200 --y 0 --z -50 --phi 0 --dry-run

# Camera frame capture
python tools/camera_capture.py --ip 192.168.4.1
python tools/camera_capture.py --scan  # Auto-detect camera on subnet

UDP Testing

# Start echo server
python tools/udp_server.py

# Send test commands
echo 'XYW:100:0:0:XZHY' | nc -u 192.168.4.1 8888
echo 'JXB:-100:10:20:30:0:0:0:0:EZHY' | nc -u 192.168.4.1 8888

Architecture Details

ESP32-S3 Dual-Core Design

• Core 0: Camera capture loop (OV2640 → JPEG)
• Core 1: WiFi streaming (MJPEG HTTP server on port 80)
• AsyncUDP: Non-blocking UDP command receiver runs on Core 1, handled via interrupt callbacks

The dual-core split ensures camera capture never blocks on WiFi transmission. UDP commands are processed asynchronously and do not interfere with streaming.

UDP Protocol

All commands are ASCII text ending with terminator (varies by command type):

# Chassis control (XYZ cartesian velocity)
XYW:<X_speed>:<Y_speed>:<W_angular>:XZHY\n

# Arm control (6 motors: height, J2-J6)
JXB:<height>:<J2>:<J3>:<J4>:<J5>:<J6>:0:0:EZHY\n

# Laser control
LASERON\n
LASEROFF\n

# Serial passthrough (any payload with ZHY or \n terminator)
<payload>ZHY\n

Critical: The ESP32 firmware parses based on terminator suffix, not command prefix.

Mechanical Arm Coordinate System & Kinematics

Coordinate frame: Base frame with Z-axis pointing UP (not down). Origin at the bottom of the J1 linear slide.

Height coordinate (d1):

• User-facing coordinate: -290 mm (bottom) to 0 mm (top)
• Physical meaning: vertical position of J2 relative to base origin
• Z-up convention: positive d1 = higher position

Planar joints (J2, J3, J4):

• All three rotate around vertical Z-axis in the XY plane
• J2 is base rotation, J3/J4 are elbow/wrist
• TCP yaw angle phi = J2 + J3 + J4 (additive)

Geometry parameters (see docs/arm.md for full derivation):

• L1 = 125 mm: J2-J3 link length
• L2 = 125 mm: J3-J4 link length
• x4 = 110 mm: J4-to-TCP horizontal offset
• z4 = 80 mm: J4-to-TCP vertical offset (variable: 55mm when gripper down, -100mm when up)

Zero offsets: Physical mechanical zero does not align with math zero (straight line):

• J2_zero = 3°, J3_zero = 7°, J4_zero = 25°
• UDP command angles = math angles + zero offsets

Inverse kinematics (tools/udp_control.py):

• Solves for joint angles given TCP pose (x, y, z, phi)
• Uses standard 2-link planar arm solution (atan2-based)
• Two solutions: --elbow-up vs default elbow-down
• Validates workspace limits and singularity checks
• See docs/arm.md for full mathematical derivation

State persistence: tools/.udp_control_state.json caches last sent joint command. This enables smooth interpolated motion from previous position without re-homing. Use --no-state-cache to disable.

ROS 2 Keyboard Backend Selection

The udp_teleop node supports three keyboard input backends:

• stdin (default on Linux/macOS): Terminal raw mode, zero dependencies, requires interactive terminal
• pynput: Cross-platform library, works in background
• win_poll: Windows-specific Win32 API polling

Backend auto-selected by platform. Override with keyboard_backend parameter in config/params.yaml.

Limitation: stdin backend fails when launched via ros2 launch because child processes lack TTY. Always use ros2 run for interactive keyboard control.

Camera Capture Tool Auto-Detection

camera_capture.py implements subnet scanning and ESP32 identification:

1. Probes common DHCP IPs on local subnet (x.x.x.1, x.x.x.100-110, etc.)
2. Verifies ESP32 by checking /status endpoint for JSON keys capture_fps and has_client
3. Falls back to full subnet scan if not found in common range
4. Connects to /stream MJPEG endpoint, parses multipart frames, extracts first valid JPEG (SOI 0xFFD8 to EOI 0xFFD9)

Use --scan to force full subnet scan, or --ip to skip detection.

Environment Setup

ROS 2 Humble via Conda (robostack)

# Create environment (one-time)
conda create -n ros2_humble -c robostack-staging -c conda-forge ros-humble-desktop
conda activate ros2_humble
conda install -c robostack-staging -c conda-forge colcon-common-extensions
pip install pynput

# Every session
conda activate ros2_humble
cd ros2
source install/setup.bash  # After first colcon build

Alternative: Native apt installation on Ubuntu 22.04 — see ros2/README.md.

PlatformIO ESP32

pip install platformio
cd jxbeye
pio pkg install  # Install dependencies

Board configuration (platformio.ini):

• Custom board definition: esp32-s3-wroom-1-n16r8
• PSRAM: Octal mode (board_build.psram_type = octal)
• Flash: 16MB QIO mode at 80MHz
• Partition table: default_16MB.csv

Important Files

Configuration

• jxbeye/platformio.ini — ESP32 build config (PSRAM settings critical)
• ros2/src/udp_teleop/config/params.yaml — ROS node parameters (IP, speeds, steps)
• tools/.udp_control_state.json — Cached arm joint state for interpolation

Documentation

• docs/arm.md — Full inverse kinematics derivation with LaTeX equations
• README.md — Project overview and quick start
• ros2/src/udp_teleop/README.md — ROS package details and keyboard mappings

Core Implementations

• jxbeye/src/main.cpp — ESP32 dual-core firmware entry point
• ros2/src/udp_teleop/udp_teleop/keyboard_control.py — ROS keyboard node
• tools/udp_control.py — Standalone arm controller with full IK solver
• tools/camera_capture.py — MJPEG stream frame extractor

Common Tasks

Change arm geometry parameters: Edit constants in tools/udp_control.py (DEFAULT_L1, DEFAULT_L2, DEFAULT_X4, DEFAULT_Z4) or pass as CLI args.

Modify joint limits: Edit DEFAULT_*_MIN/MAX in tools/udp_control.py or use --height-min, --j2-max, etc.

Adjust interpolation smoothness: Change --duration (total time) and --rate (Hz) in udp_control.py. Default is 1.0s at 20Hz = 20 steps.

Debug UDP protocol: Use tools/udp_server.py as echo server, point ROS/tools at 127.0.0.1:8888 to inspect raw commands.

Test kinematics without hardware: Use --dry-run flag with udp_control.py to print UDP commands without sending.

Verify IK correctness: Use --show-fk flag to compute forward kinematics of the solved joint angles and compare to target pose.

Dataset

The dataset.zip and dataset/ directory contain competition-specific training data (exact format unknown from structure alone).