diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..f3056a3 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,242 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +CRAIC (Camera-Robot AI Control System) — Competition code for the China Robot and Artificial Intelligence Competition (中国机器人及人工智能大赛), Robot Task Challenge (Small Desktop Level). + +**Hardware**: ESP32-S3-WROOM-1-N16R8 with OV2640 camera, 6-DOF mechanical arm with Feettech SCS/STS serial servos. + +**Architecture**: Three-tier system communicating via UDP port 8888: +1. **ESP32-S3 firmware** (`jxbeye/`) — Dual-core camera streaming + UDP command receiver +2. **ROS 2 teleop** (`ros2/src/udp_teleop/`) — Keyboard control node for chassis and arm +3. **Python tools** (`tools/`) — Standalone control scripts with inverse kinematics + +## Build & Run Commands + +### ESP32-S3 Firmware + +```bash +cd jxbeye +pio run -t upload # Build and flash firmware +pio device monitor # Serial monitor (1000000 baud) +``` + +**First boot**: ESP32 creates AP `ESP32-S3-Camera` (password `12345678`), access web UI at `http://192.168.4.1`. + +**WiFi configuration via serial**: Send `WIFI:SSID:PASSWORD` to configure station mode. + +### ROS 2 Teleop + +```bash +# Build (from ros2/ directory) +conda activate ros2_humble +colcon build --symlink-install --packages-select udp_teleop +source install/setup.bash + +# Run keyboard control +ros2 run udp_teleop keyboard_control \ + --ros-args --params-file src/udp_teleop/config/params.yaml + +# Override target IP +ros2 run udp_teleop keyboard_control \ + --ros-args -p udp_ip:=192.168.4.1 -p udp_port:=8888 +``` + +**Keyboard mappings**: +- Chassis: W/S (forward/back), A/D (strafe), Q/E (rotate) +- Arm: ↑/↓ (height), 2-6 (select joint), ←/→ (adjust angle) + +**Important**: Must use `ros2 run`, not `ros2 launch` — the `stdin` keyboard backend requires an interactive terminal. + +### Python Arm Control Tools + +```bash +# Direct joint command with interpolation +python tools/udp_control.py joints \ + --height -100 --j2 10 --j3 20 --j4 30 \ + --duration 1.0 --rate 20 + +# Cartesian pose mode (uses inverse kinematics) +python tools/udp_control.py pose \ + --x 150 --y 50 --z -100 --phi 45 \ + --duration 1.0 + +# Dry run (print commands without sending) +python tools/udp_control.py pose --x 200 --y 0 --z -50 --phi 0 --dry-run + +# Camera frame capture +python tools/camera_capture.py --ip 192.168.4.1 +python tools/camera_capture.py --scan # Auto-detect camera on subnet +``` + +### UDP Testing + +```bash +# Start echo server +python tools/udp_server.py + +# Send test commands +echo 'XYW:100:0:0:XZHY' | nc -u 192.168.4.1 8888 +echo 'JXB:-100:10:20:30:0:0:0:0:EZHY' | nc -u 192.168.4.1 8888 +``` + +## Architecture Details + +### ESP32-S3 Dual-Core Design + +- **Core 0**: Camera capture loop (OV2640 → JPEG) +- **Core 1**: WiFi streaming (MJPEG HTTP server on port 80) +- **AsyncUDP**: Non-blocking UDP command receiver runs on Core 1, handled via interrupt callbacks + +The dual-core split ensures camera capture never blocks on WiFi transmission. UDP commands are processed asynchronously and do not interfere with streaming. + +### UDP Protocol + +All commands are ASCII text ending with terminator (varies by command type): + +``` +# Chassis control (XYZ cartesian velocity) +XYW::::XZHY\n + +# Arm control (6 motors: height, J2-J6) +JXB:::::::0:0:EZHY\n + +# Laser control +LASERON\n +LASEROFF\n + +# Serial passthrough (any payload with ZHY or \n terminator) +ZHY\n +``` + +**Critical**: The ESP32 firmware parses based on terminator suffix, not command prefix. + +### Mechanical Arm Coordinate System & Kinematics + +**Coordinate frame**: Base frame with Z-axis pointing UP (not down). Origin at the bottom of the J1 linear slide. + +**Height coordinate (d1)**: +- User-facing coordinate: `-290 mm` (bottom) to `0 mm` (top) +- Physical meaning: vertical position of J2 relative to base origin +- **Z-up convention**: positive d1 = higher position + +**Planar joints (J2, J3, J4)**: +- All three rotate around vertical Z-axis in the XY plane +- J2 is base rotation, J3/J4 are elbow/wrist +- TCP yaw angle `phi = J2 + J3 + J4` (additive) + +**Geometry parameters** (see `docs/arm.md` for full derivation): +- `L1 = 125 mm`: J2-J3 link length +- `L2 = 125 mm`: J3-J4 link length +- `x4 = 110 mm`: J4-to-TCP horizontal offset +- `z4 = 80 mm`: J4-to-TCP vertical offset (variable: 55mm when gripper down, -100mm when up) + +**Zero offsets**: Physical mechanical zero does not align with math zero (straight line): +- `J2_zero = 3°`, `J3_zero = 7°`, `J4_zero = 25°` +- UDP command angles = math angles + zero offsets + +**Inverse kinematics** (`tools/udp_control.py`): +- Solves for joint angles given TCP pose `(x, y, z, phi)` +- Uses standard 2-link planar arm solution (atan2-based) +- Two solutions: `--elbow-up` vs default elbow-down +- Validates workspace limits and singularity checks +- See `docs/arm.md` for full mathematical derivation + +**State persistence**: `tools/.udp_control_state.json` caches last sent joint command. This enables smooth interpolated motion from previous position without re-homing. Use `--no-state-cache` to disable. + +### ROS 2 Keyboard Backend Selection + +The `udp_teleop` node supports three keyboard input backends: + +- `stdin` (default on Linux/macOS): Terminal raw mode, zero dependencies, **requires interactive terminal** +- `pynput`: Cross-platform library, works in background +- `win_poll`: Windows-specific Win32 API polling + +Backend auto-selected by platform. Override with `keyboard_backend` parameter in `config/params.yaml`. + +**Limitation**: `stdin` backend fails when launched via `ros2 launch` because child processes lack TTY. Always use `ros2 run` for interactive keyboard control. + +### Camera Capture Tool Auto-Detection + +`camera_capture.py` implements subnet scanning and ESP32 identification: + +1. Probes common DHCP IPs on local subnet (`x.x.x.1`, `x.x.x.100-110`, etc.) +2. Verifies ESP32 by checking `/status` endpoint for JSON keys `capture_fps` and `has_client` +3. Falls back to full subnet scan if not found in common range +4. Connects to `/stream` MJPEG endpoint, parses multipart frames, extracts first valid JPEG (SOI `0xFFD8` to EOI `0xFFD9`) + +Use `--scan` to force full subnet scan, or `--ip` to skip detection. + +## Environment Setup + +### ROS 2 Humble via Conda (robostack) + +```bash +# Create environment (one-time) +conda create -n ros2_humble -c robostack-staging -c conda-forge ros-humble-desktop +conda activate ros2_humble +conda install -c robostack-staging -c conda-forge colcon-common-extensions +pip install pynput + +# Every session +conda activate ros2_humble +cd ros2 +source install/setup.bash # After first colcon build +``` + +**Alternative**: Native apt installation on Ubuntu 22.04 — see `ros2/README.md`. + +### PlatformIO ESP32 + +```bash +pip install platformio +cd jxbeye +pio pkg install # Install dependencies +``` + +**Board configuration** (`platformio.ini`): +- Custom board definition: `esp32-s3-wroom-1-n16r8` +- PSRAM: Octal mode (`board_build.psram_type = octal`) +- Flash: 16MB QIO mode at 80MHz +- Partition table: `default_16MB.csv` + +## Important Files + +### Configuration + +- `jxbeye/platformio.ini` — ESP32 build config (PSRAM settings critical) +- `ros2/src/udp_teleop/config/params.yaml` — ROS node parameters (IP, speeds, steps) +- `tools/.udp_control_state.json` — Cached arm joint state for interpolation + +### Documentation + +- `docs/arm.md` — Full inverse kinematics derivation with LaTeX equations +- `README.md` — Project overview and quick start +- `ros2/src/udp_teleop/README.md` — ROS package details and keyboard mappings + +### Core Implementations + +- `jxbeye/src/main.cpp` — ESP32 dual-core firmware entry point +- `ros2/src/udp_teleop/udp_teleop/keyboard_control.py` — ROS keyboard node +- `tools/udp_control.py` — Standalone arm controller with full IK solver +- `tools/camera_capture.py` — MJPEG stream frame extractor + +## Common Tasks + +**Change arm geometry parameters**: Edit constants in `tools/udp_control.py` (`DEFAULT_L1`, `DEFAULT_L2`, `DEFAULT_X4`, `DEFAULT_Z4`) or pass as CLI args. + +**Modify joint limits**: Edit `DEFAULT_*_MIN/MAX` in `tools/udp_control.py` or use `--height-min`, `--j2-max`, etc. + +**Adjust interpolation smoothness**: Change `--duration` (total time) and `--rate` (Hz) in `udp_control.py`. Default is 1.0s at 20Hz = 20 steps. + +**Debug UDP protocol**: Use `tools/udp_server.py` as echo server, point ROS/tools at `127.0.0.1:8888` to inspect raw commands. + +**Test kinematics without hardware**: Use `--dry-run` flag with `udp_control.py` to print UDP commands without sending. + +**Verify IK correctness**: Use `--show-fk` flag to compute forward kinematics of the solved joint angles and compare to target pose. + +## Dataset + +The `dataset.zip` and `dataset/` directory contain competition-specific training data (exact format unknown from structure alone).