243 lines
8.7 KiB
Markdown
243 lines
8.7 KiB
Markdown
# CLAUDE.md
|
|
|
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
|
|
## Project Overview
|
|
|
|
CRAIC (Camera-Robot AI Control System) — Competition code for the China Robot and Artificial Intelligence Competition (中国机器人及人工智能大赛), Robot Task Challenge (Small Desktop Level).
|
|
|
|
**Hardware**: ESP32-S3-WROOM-1-N16R8 with OV2640 camera, 6-DOF mechanical arm with Feettech SCS/STS serial servos.
|
|
|
|
**Architecture**: Three-tier system communicating via UDP port 8888:
|
|
1. **ESP32-S3 firmware** (`jxbeye/`) — Dual-core camera streaming + UDP command receiver
|
|
2. **ROS 2 teleop** (`ros2/src/udp_teleop/`) — Keyboard control node for chassis and arm
|
|
3. **Python tools** (`tools/`) — Standalone control scripts with inverse kinematics
|
|
|
|
## Build & Run Commands
|
|
|
|
### ESP32-S3 Firmware
|
|
|
|
```bash
|
|
cd jxbeye
|
|
pio run -t upload # Build and flash firmware
|
|
pio device monitor # Serial monitor (1000000 baud)
|
|
```
|
|
|
|
**First boot**: ESP32 creates AP `ESP32-S3-Camera` (password `12345678`), access web UI at `http://192.168.4.1`.
|
|
|
|
**WiFi configuration via serial**: Send `WIFI:SSID:PASSWORD` to configure station mode.
|
|
|
|
### ROS 2 Teleop
|
|
|
|
```bash
|
|
# Build (from ros2/ directory)
|
|
conda activate ros2_humble
|
|
colcon build --symlink-install --packages-select udp_teleop
|
|
source install/setup.bash
|
|
|
|
# Run keyboard control
|
|
ros2 run udp_teleop keyboard_control \
|
|
--ros-args --params-file src/udp_teleop/config/params.yaml
|
|
|
|
# Override target IP
|
|
ros2 run udp_teleop keyboard_control \
|
|
--ros-args -p udp_ip:=192.168.4.1 -p udp_port:=8888
|
|
```
|
|
|
|
**Keyboard mappings**:
|
|
- Chassis: W/S (forward/back), A/D (strafe), Q/E (rotate)
|
|
- Arm: ↑/↓ (height), 2-6 (select joint), ←/→ (adjust angle)
|
|
|
|
**Important**: Must use `ros2 run`, not `ros2 launch` — the `stdin` keyboard backend requires an interactive terminal.
|
|
|
|
### Python Arm Control Tools
|
|
|
|
```bash
|
|
# Direct joint command with interpolation
|
|
python tools/udp_control.py joints \
|
|
--height -100 --j2 10 --j3 20 --j4 30 \
|
|
--duration 1.0 --rate 20
|
|
|
|
# Cartesian pose mode (uses inverse kinematics)
|
|
python tools/udp_control.py pose \
|
|
--x 150 --y 50 --z -100 --phi 45 \
|
|
--duration 1.0
|
|
|
|
# Dry run (print commands without sending)
|
|
python tools/udp_control.py pose --x 200 --y 0 --z -50 --phi 0 --dry-run
|
|
|
|
# Camera frame capture
|
|
python tools/camera_capture.py --ip 192.168.4.1
|
|
python tools/camera_capture.py --scan # Auto-detect camera on subnet
|
|
```
|
|
|
|
### UDP Testing
|
|
|
|
```bash
|
|
# Start echo server
|
|
python tools/udp_server.py
|
|
|
|
# Send test commands
|
|
echo 'XYW:100:0:0:XZHY' | nc -u 192.168.4.1 8888
|
|
echo 'JXB:-100:10:20:30:0:0:0:0:EZHY' | nc -u 192.168.4.1 8888
|
|
```
|
|
|
|
## Architecture Details
|
|
|
|
### ESP32-S3 Dual-Core Design
|
|
|
|
- **Core 0**: Camera capture loop (OV2640 → JPEG)
|
|
- **Core 1**: WiFi streaming (MJPEG HTTP server on port 80)
|
|
- **AsyncUDP**: Non-blocking UDP command receiver runs on Core 1, handled via interrupt callbacks
|
|
|
|
The dual-core split ensures camera capture never blocks on WiFi transmission. UDP commands are processed asynchronously and do not interfere with streaming.
|
|
|
|
### UDP Protocol
|
|
|
|
All commands are ASCII text ending with terminator (varies by command type):
|
|
|
|
```
|
|
# Chassis control (XYZ cartesian velocity)
|
|
XYW:<X_speed>:<Y_speed>:<W_angular>:XZHY\n
|
|
|
|
# Arm control (6 motors: height, J2-J6)
|
|
JXB:<height>:<J2>:<J3>:<J4>:<J5>:<J6>:0:0:EZHY\n
|
|
|
|
# Laser control
|
|
LASERON\n
|
|
LASEROFF\n
|
|
|
|
# Serial passthrough (any payload with ZHY or \n terminator)
|
|
<payload>ZHY\n
|
|
```
|
|
|
|
**Critical**: The ESP32 firmware parses based on terminator suffix, not command prefix.
|
|
|
|
### Mechanical Arm Coordinate System & Kinematics
|
|
|
|
**Coordinate frame**: Base frame with Z-axis pointing UP (not down). Origin at the bottom of the J1 linear slide.
|
|
|
|
**Height coordinate (d1)**:
|
|
- User-facing coordinate: `-290 mm` (bottom) to `0 mm` (top)
|
|
- Physical meaning: vertical position of J2 relative to base origin
|
|
- **Z-up convention**: positive d1 = higher position
|
|
|
|
**Planar joints (J2, J3, J4)**:
|
|
- All three rotate around vertical Z-axis in the XY plane
|
|
- J2 is base rotation, J3/J4 are elbow/wrist
|
|
- TCP yaw angle `phi = J2 + J3 + J4` (additive)
|
|
|
|
**Geometry parameters** (see `docs/arm.md` for full derivation):
|
|
- `L1 = 125 mm`: J2-J3 link length
|
|
- `L2 = 125 mm`: J3-J4 link length
|
|
- `x4 = 110 mm`: J4-to-TCP horizontal offset
|
|
- `z4 = 80 mm`: J4-to-TCP vertical offset (variable: 55mm when gripper down, -100mm when up)
|
|
|
|
**Zero offsets**: Physical mechanical zero does not align with math zero (straight line):
|
|
- `J2_zero = 3°`, `J3_zero = 7°`, `J4_zero = 25°`
|
|
- UDP command angles = math angles + zero offsets
|
|
|
|
**Inverse kinematics** (`tools/udp_control.py`):
|
|
- Solves for joint angles given TCP pose `(x, y, z, phi)`
|
|
- Uses standard 2-link planar arm solution (atan2-based)
|
|
- Two solutions: `--elbow-up` vs default elbow-down
|
|
- Validates workspace limits and singularity checks
|
|
- See `docs/arm.md` for full mathematical derivation
|
|
|
|
**State persistence**: `tools/.udp_control_state.json` caches last sent joint command. This enables smooth interpolated motion from previous position without re-homing. Use `--no-state-cache` to disable.
|
|
|
|
### ROS 2 Keyboard Backend Selection
|
|
|
|
The `udp_teleop` node supports three keyboard input backends:
|
|
|
|
- `stdin` (default on Linux/macOS): Terminal raw mode, zero dependencies, **requires interactive terminal**
|
|
- `pynput`: Cross-platform library, works in background
|
|
- `win_poll`: Windows-specific Win32 API polling
|
|
|
|
Backend auto-selected by platform. Override with `keyboard_backend` parameter in `config/params.yaml`.
|
|
|
|
**Limitation**: `stdin` backend fails when launched via `ros2 launch` because child processes lack TTY. Always use `ros2 run` for interactive keyboard control.
|
|
|
|
### Camera Capture Tool Auto-Detection
|
|
|
|
`camera_capture.py` implements subnet scanning and ESP32 identification:
|
|
|
|
1. Probes common DHCP IPs on local subnet (`x.x.x.1`, `x.x.x.100-110`, etc.)
|
|
2. Verifies ESP32 by checking `/status` endpoint for JSON keys `capture_fps` and `has_client`
|
|
3. Falls back to full subnet scan if not found in common range
|
|
4. Connects to `/stream` MJPEG endpoint, parses multipart frames, extracts first valid JPEG (SOI `0xFFD8` to EOI `0xFFD9`)
|
|
|
|
Use `--scan` to force full subnet scan, or `--ip` to skip detection.
|
|
|
|
## Environment Setup
|
|
|
|
### ROS 2 Humble via Conda (robostack)
|
|
|
|
```bash
|
|
# Create environment (one-time)
|
|
conda create -n ros2_humble -c robostack-staging -c conda-forge ros-humble-desktop
|
|
conda activate ros2_humble
|
|
conda install -c robostack-staging -c conda-forge colcon-common-extensions
|
|
pip install pynput
|
|
|
|
# Every session
|
|
conda activate ros2_humble
|
|
cd ros2
|
|
source install/setup.bash # After first colcon build
|
|
```
|
|
|
|
**Alternative**: Native apt installation on Ubuntu 22.04 — see `ros2/README.md`.
|
|
|
|
### PlatformIO ESP32
|
|
|
|
```bash
|
|
pip install platformio
|
|
cd jxbeye
|
|
pio pkg install # Install dependencies
|
|
```
|
|
|
|
**Board configuration** (`platformio.ini`):
|
|
- Custom board definition: `esp32-s3-wroom-1-n16r8`
|
|
- PSRAM: Octal mode (`board_build.psram_type = octal`)
|
|
- Flash: 16MB QIO mode at 80MHz
|
|
- Partition table: `default_16MB.csv`
|
|
|
|
## Important Files
|
|
|
|
### Configuration
|
|
|
|
- `jxbeye/platformio.ini` — ESP32 build config (PSRAM settings critical)
|
|
- `ros2/src/udp_teleop/config/params.yaml` — ROS node parameters (IP, speeds, steps)
|
|
- `tools/.udp_control_state.json` — Cached arm joint state for interpolation
|
|
|
|
### Documentation
|
|
|
|
- `docs/arm.md` — Full inverse kinematics derivation with LaTeX equations
|
|
- `README.md` — Project overview and quick start
|
|
- `ros2/src/udp_teleop/README.md` — ROS package details and keyboard mappings
|
|
|
|
### Core Implementations
|
|
|
|
- `jxbeye/src/main.cpp` — ESP32 dual-core firmware entry point
|
|
- `ros2/src/udp_teleop/udp_teleop/keyboard_control.py` — ROS keyboard node
|
|
- `tools/udp_control.py` — Standalone arm controller with full IK solver
|
|
- `tools/camera_capture.py` — MJPEG stream frame extractor
|
|
|
|
## Common Tasks
|
|
|
|
**Change arm geometry parameters**: Edit constants in `tools/udp_control.py` (`DEFAULT_L1`, `DEFAULT_L2`, `DEFAULT_X4`, `DEFAULT_Z4`) or pass as CLI args.
|
|
|
|
**Modify joint limits**: Edit `DEFAULT_*_MIN/MAX` in `tools/udp_control.py` or use `--height-min`, `--j2-max`, etc.
|
|
|
|
**Adjust interpolation smoothness**: Change `--duration` (total time) and `--rate` (Hz) in `udp_control.py`. Default is 1.0s at 20Hz = 20 steps.
|
|
|
|
**Debug UDP protocol**: Use `tools/udp_server.py` as echo server, point ROS/tools at `127.0.0.1:8888` to inspect raw commands.
|
|
|
|
**Test kinematics without hardware**: Use `--dry-run` flag with `udp_control.py` to print UDP commands without sending.
|
|
|
|
**Verify IK correctness**: Use `--show-fk` flag to compute forward kinematics of the solved joint angles and compare to target pose.
|
|
|
|
## Dataset
|
|
|
|
The `dataset.zip` and `dataset/` directory contain competition-specific training data (exact format unknown from structure alone).
|