# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

CRAIC (Camera-Robot AI Control System) — Competition code for the China Robot and Artificial Intelligence Competition (中国机器人及人工智能大赛), Robot Task Challenge (Small Desktop Level).

**Hardware**: ESP32-S3-WROOM-1-N16R8 with OV2640 camera, 6-DOF mechanical arm with Feettech SCS/STS serial servos.

**Architecture**: Three-tier system communicating via UDP port 8888:
1. **ESP32-S3 firmware** (`jxbeye/`) — Dual-core camera streaming + UDP command receiver
2. **ROS 2 teleop** (`ros2/src/udp_teleop/`) — Keyboard control node for chassis and arm
3. **Python tools** (`tools/`) — Standalone control scripts with inverse kinematics

## Build & Run Commands

### ESP32-S3 Firmware

```bash
cd jxbeye
pio run -t upload          # Build and flash firmware
pio device monitor         # Serial monitor (1000000 baud)
```

**First boot**: ESP32 creates AP `ESP32-S3-Camera` (password `12345678`), access web UI at `http://192.168.4.1`.

**WiFi configuration via serial**: Send `WIFI:SSID:PASSWORD` to configure station mode.

### ROS 2 Teleop

```bash
# Build (from ros2/ directory)
conda activate ros2_humble
colcon build --symlink-install --packages-select udp_teleop
source install/setup.bash

# Run keyboard control
ros2 run udp_teleop keyboard_control \
    --ros-args --params-file src/udp_teleop/config/params.yaml

# Override target IP
ros2 run udp_teleop keyboard_control \
    --ros-args -p udp_ip:=192.168.4.1 -p udp_port:=8888
```

**Keyboard mappings**:
- Chassis: W/S (forward/back), A/D (strafe), Q/E (rotate)
- Arm: ↑/↓ (height), 2-6 (select joint), ←/→ (adjust angle)

**Important**: Must use `ros2 run`, not `ros2 launch` — the `stdin` keyboard backend requires an interactive terminal.

### Python Arm Control Tools

```bash
# Direct joint command with interpolation
python tools/udp_control.py joints \
    --height -100 --j2 10 --j3 20 --j4 30 \
    --duration 1.0 --rate 20

# Cartesian pose mode (uses inverse kinematics)
python tools/udp_control.py pose \
    --x 150 --y 50 --z -100 --phi 45 \
    --duration 1.0

# Dry run (print commands without sending)
python tools/udp_control.py pose --x 200 --y 0 --z -50 --phi 0 --dry-run

# Camera frame capture
python tools/camera_capture.py --ip 192.168.4.1
python tools/camera_capture.py --scan  # Auto-detect camera on subnet
```

### UDP Testing

```bash
# Start echo server
python tools/udp_server.py

# Send test commands
echo 'XYW:100:0:0:XZHY' | nc -u 192.168.4.1 8888
echo 'JXB:-100:10:20:30:0:0:0:0:EZHY' | nc -u 192.168.4.1 8888
```

## Architecture Details

### ESP32-S3 Dual-Core Design

- **Core 0**: Camera capture loop (OV2640 → JPEG)
- **Core 1**: WiFi streaming (MJPEG HTTP server on port 80)
- **AsyncUDP**: Non-blocking UDP command receiver runs on Core 1, handled via interrupt callbacks

The dual-core split ensures camera capture never blocks on WiFi transmission. UDP commands are processed asynchronously and do not interfere with streaming.

### UDP Protocol

All commands are ASCII text ending with terminator (varies by command type):

```
# Chassis control (XYZ cartesian velocity)
XYW:<X_speed>:<Y_speed>:<W_angular>:XZHY\n

# Arm control (6 motors: height, J2-J6)
JXB:<height>:<J2>:<J3>:<J4>:<J5>:<J6>:0:0:EZHY\n

# Laser control
LASERON\n
LASEROFF\n

# Serial passthrough (any payload with ZHY or \n terminator)
<payload>ZHY\n
```

**Critical**: The ESP32 firmware parses based on terminator suffix, not command prefix.

### Mechanical Arm Coordinate System & Kinematics

**Coordinate frame**: Base frame with Z-axis pointing UP (not down). Origin at the bottom of the J1 linear slide.

**Height coordinate (d1)**:
- User-facing coordinate: `-290 mm` (bottom) to `0 mm` (top)
- Physical meaning: vertical position of J2 relative to base origin
- **Z-up convention**: positive d1 = higher position

**Planar joints (J2, J3, J4)**:
- All three rotate around vertical Z-axis in the XY plane
- J2 is base rotation, J3/J4 are elbow/wrist
- TCP yaw angle `phi = J2 + J3 + J4` (additive)

**Geometry parameters** (see `docs/arm.md` for full derivation):
- `L1 = 125 mm`: J2-J3 link length
- `L2 = 125 mm`: J3-J4 link length  
- `x4 = 110 mm`: J4-to-TCP horizontal offset
- `z4 = 80 mm`: J4-to-TCP vertical offset (variable: 55mm when gripper down, -100mm when up)

**Zero offsets**: Physical mechanical zero does not align with math zero (straight line):
- `J2_zero = 3°`, `J3_zero = 7°`, `J4_zero = 25°`
- UDP command angles = math angles + zero offsets

**Inverse kinematics** (`tools/udp_control.py`):
- Solves for joint angles given TCP pose `(x, y, z, phi)`
- Uses standard 2-link planar arm solution (atan2-based)
- Two solutions: `--elbow-up` vs default elbow-down
- Validates workspace limits and singularity checks
- See `docs/arm.md` for full mathematical derivation

**State persistence**: `tools/.udp_control_state.json` caches last sent joint command. This enables smooth interpolated motion from previous position without re-homing. Use `--no-state-cache` to disable.

### ROS 2 Keyboard Backend Selection

The `udp_teleop` node supports three keyboard input backends:

- `stdin` (default on Linux/macOS): Terminal raw mode, zero dependencies, **requires interactive terminal**
- `pynput`: Cross-platform library, works in background
- `win_poll`: Windows-specific Win32 API polling

Backend auto-selected by platform. Override with `keyboard_backend` parameter in `config/params.yaml`.

**Limitation**: `stdin` backend fails when launched via `ros2 launch` because child processes lack TTY. Always use `ros2 run` for interactive keyboard control.

### Camera Capture Tool Auto-Detection

`camera_capture.py` implements subnet scanning and ESP32 identification:

1. Probes common DHCP IPs on local subnet (`x.x.x.1`, `x.x.x.100-110`, etc.)
2. Verifies ESP32 by checking `/status` endpoint for JSON keys `capture_fps` and `has_client`
3. Falls back to full subnet scan if not found in common range
4. Connects to `/stream` MJPEG endpoint, parses multipart frames, extracts first valid JPEG (SOI `0xFFD8` to EOI `0xFFD9`)

Use `--scan` to force full subnet scan, or `--ip` to skip detection.

## Environment Setup

### ROS 2 Humble via Conda (robostack)

```bash
# Create environment (one-time)
conda create -n ros2_humble -c robostack-staging -c conda-forge ros-humble-desktop
conda activate ros2_humble
conda install -c robostack-staging -c conda-forge colcon-common-extensions
pip install pynput

# Every session
conda activate ros2_humble
cd ros2
source install/setup.bash  # After first colcon build
```

**Alternative**: Native apt installation on Ubuntu 22.04 — see `ros2/README.md`.

### PlatformIO ESP32

```bash
pip install platformio
cd jxbeye
pio pkg install  # Install dependencies
```

**Board configuration** (`platformio.ini`):
- Custom board definition: `esp32-s3-wroom-1-n16r8`
- PSRAM: Octal mode (`board_build.psram_type = octal`)
- Flash: 16MB QIO mode at 80MHz
- Partition table: `default_16MB.csv`

## Important Files

### Configuration

- `jxbeye/platformio.ini` — ESP32 build config (PSRAM settings critical)
- `ros2/src/udp_teleop/config/params.yaml` — ROS node parameters (IP, speeds, steps)
- `tools/.udp_control_state.json` — Cached arm joint state for interpolation

### Documentation

- `docs/arm.md` — Full inverse kinematics derivation with LaTeX equations
- `README.md` — Project overview and quick start
- `ros2/src/udp_teleop/README.md` — ROS package details and keyboard mappings

### Core Implementations

- `jxbeye/src/main.cpp` — ESP32 dual-core firmware entry point
- `ros2/src/udp_teleop/udp_teleop/keyboard_control.py` — ROS keyboard node
- `tools/udp_control.py` — Standalone arm controller with full IK solver
- `tools/camera_capture.py` — MJPEG stream frame extractor

## Common Tasks

**Change arm geometry parameters**: Edit constants in `tools/udp_control.py` (`DEFAULT_L1`, `DEFAULT_L2`, `DEFAULT_X4`, `DEFAULT_Z4`) or pass as CLI args.

**Modify joint limits**: Edit `DEFAULT_*_MIN/MAX` in `tools/udp_control.py` or use `--height-min`, `--j2-max`, etc.

**Adjust interpolation smoothness**: Change `--duration` (total time) and `--rate` (Hz) in `udp_control.py`. Default is 1.0s at 20Hz = 20 steps.

**Debug UDP protocol**: Use `tools/udp_server.py` as echo server, point ROS/tools at `127.0.0.1:8888` to inspect raw commands.

**Test kinematics without hardware**: Use `--dry-run` flag with `udp_control.py` to print UDP commands without sending.

**Verify IK correctness**: Use `--show-fk` flag to compute forward kinematics of the solved joint angles and compare to target pose.

## Dataset

The `dataset.zip` and `dataset/` directory contain competition-specific training data (exact format unknown from structure alone).