UR10e Zero-Shot Package Management System (ZSPMS)

A Webots-based autonomous package sorting system using a UR10e robotic arm, YOLO-based package detection, OCR + CLIP verification, and a joint-sensor-driven finite state machine (FSM) for reliable pick-and-place operations.

This project demonstrates a practical end-to-end robotics pipeline: perception → reasoning → manipulation → supervision.

---

📌 Key Features

• Custom-trained YOLOv8 package detector (no suitable pretrained model existed)
• Webots Supervisor-based simulation
• Single-package perception pipeline (robust against ID duplication)
• OCR + fuzzy matching + CLIP verification to reduce false label reads
• 6-DOF UR10e joint control (no IK, no hallucinated APIs)
• Joint-sensor–driven FSM (no timers, no heuristics)
• Conveyor belt control synchronized with robot actions
• Dynamic package spawning with safe spacing
• Extensible architecture (GUI overlay, analytics, multi-arm support)

---

🎯 Motivation & Problem Statement

There is no reliable pretrained computer vision model that can:

• Detect generic cardboard packages
• Work inside Webots
• Support industrial sorting logic

Existing object detectors (COCO, OpenImages, etc.) do not include a “package” class suitable for logistics scenarios.

Therefore, a custom dataset and training pipeline was required.

---

🧠 Dataset Creation (Important)

❗ No Pretrained Model for Packages

There is no pretrained YOLO model that reliably detects shipping packages. As a result, a custom dataset had to be created inside Webots.

---

🏗️ Data Collection in Webots

A custom Webots controller was written to:

1. Capture RGB images from the camera
2. Save bounding boxes / segmentation labels
3. Export images + annotations to disk

These images were then prepared for training using Ultralytics YOLOv8 segmentation format.

---

📦 Ultralytics Dataset Structure

Dataset format followed:

📎 https://docs.ultralytics.com/datasets/segment/package-seg/

package-seg/
├── images/
│   ├── train/
│   ├── val/
├── labels/
│   ├── train/
│   ├── val/
├── dataset.yaml

---

⚠️ Important Training Detail (Critical)

Ultralytics does not download required files until the first training run.

To resolve this:

1. A single-epoch training run was executed:

   yolo segment train model=yolov8n-seg.pt data=dataset.yaml epochs=1

2. This triggered:

   • Automatic directory creation
   • Cache setup
   • Metadata generation

3. After that, custom Webots images and labels were added into:

   images/train
   labels/train
   

4. Training was then resumed normally.

---

🏋️ Training

After data insertion:

yolo segment train \
  model=yolov8n-seg.pt \
  data=dataset.yaml \
  epochs=50 \
  imgsz=640

The resulting model is used as:

best.pt

---

🧩 System Architecture

┌───────────┐
│ Conveyor  │
└─────┬─────┘
      │
┌─────▼─────┐
│  Camera   │
│ (Webots)  │
└─────┬─────┘
      │
┌─────▼───────────┐
│ YOLOv8 Detector │  ← custom-trained
└─────┬───────────┘
      │
┌─────▼───────────────┐
│ OCR (EasyOCR)       │
│ + Fuzzy Matching    │
│ + CLIP Verification│
└─────┬───────────────┘
      │
┌─────▼───────────────┐
│ Destination Lock    │
└─────┬───────────────┘
      │
┌─────▼───────────────┐
│ PickFSM (Joint-based│
│ Finite State Machine│
└─────┬───────────────┘
      │
┌─────▼──────────┐
│ UR10e Arm      │
│ (6-DOF joints) │
└─────┬──────────┘
      │
┌─────▼──────────┐
│ Plastic Crates │
│ (Destinations) │
└────────────────┘

---

🤖 Robotic Control (Important Design Choices)

❌ No Built-in IK in Webots UR10e

Webots R2023b:

• ❌ No inverseKinematics
• ❌ No endEffector field
• ❌ No children field on UR10e PROTO

➡️ All arm motion is controlled directly via joint motors. su

✅ Joint-Sensor–Driven FSM (Correct Approach)

Instead of:

• Timers ❌
• Wrist heuristics ❌
• Magic constants ❌

I used:

• All 6 joint sensors
• Target joint convergence
• Distance sensor for release confirmation

---

🧠 PickFSM States

IDLE
 → GRASP
 → MOVE_TO_BIN (wait until ALL joints reach target)
 → RELEASE (open gripper)
 → WAIT_RELEASE_CLEAR (distance sensor)
 → MOVE_HOME (wait until ALL joints reach HOME)
 → IDLE

✔ Deterministic ✔ Physics-safe ✔ Conveyor synchronized

---

🚚 Conveyor Belt Logic

• RUN while scanning and detecting
• STOP immediately when pick starts
• RESUME only after arm returns HOME

This ensures:

• No missed packages
• Stable OCR
• Safe manipulation

---

📂 Configuration

bins.json (Optional)

{
  "london":     [3.9, -0.8, 0.50, -2.50, -1.28, 0.05],
  "manchester": [3.5, -0.8, 0.50, -0.20, -1.48, 0.05],
  "sheffield":  [0.02, -1.78, -1.90, -1.55, 1.52, -0.01],
  "leeds":      [2.2, -0.8, 0.50, -1.30, -1.48, 0.05],
  "lagos":      [1.7, -0.4, 0.50, -2.50, -1.28, 0.05]
}

Overrides default joint targets.

---

🧪 Dependencies

• Webots R2023b
• Python 3.9+
• Ultralytics YOLOv8
• OpenCV
• EasyOCR
• PyTorch
• Transformers (CLIP)

pip install ultralytics opencv-python easyocr torch transformers

---

🔮 Future Work

• GUI overlay (counts per destination, last OCR text)
• Multi-package tracking
• Dynamic bin calibration
• Real robot deployment (ROS 2 bridge)
• Vision-language grounding improvements

---

📜 License

MIT License

---

👤 Author

Abiola Oluwaseun Onasanya
MSc Artificial Intelligence - Advanced Robotics and Artificial Intelligence
Sheffield Hallam University

---