Computer Vision for Robotics

Session activity

Work through the class notebook: load an image, inspect its pixels as a NumPy array, convert between color spaces, transform it, draw on it and adjust brightness and contrast.

Learning objectives

• Understand how images are represented digitally and perform basic image manipulation using OpenCV.

Key concepts introduced

• Introduction to computer vision: applications in robotics
• Digital images: pixels, resolution, channels
• Image representation using NumPy
• Color spaces: BGR, RGB, grayscale, HSV
• Reading and displaying images
• Image transformations: resize, crop, rotate, flip
• Drawing on images: lines, rectangles, circles, text
• Basic image enhancement: brightness, contrast
• OpenCV functions

Session activity

Open a webcam stream, capture frames, threshold them, find contours and features, and track an object by its color or contour.

Learning objectives

• Learn how computers extract meaningful information from images without deep learning.

Key concepts introduced

• Camera fundamentals: webcam access, video streams, capturing frames
• Image preprocessing: image filtering
• Thresholding: binary, adaptive, Otsu, HSV color
• Feature extraction: contours, image moments, ORB features, feature matching
• Classical object tracking: color tracking, contour tracking, centroid estimation

Session activity

Build and train a first multi-layer perceptron image classifier in PyTorch, then evaluate it with accuracy, precision, recall and a confusion matrix.

Learning objectives

• Understand how neural networks learn and train your first image classifier.

Key concepts introduced

• Introduction: why deep learning, supervised learning, what it means to train a model, datasets and labels
• PyTorch basics: tensors, tensor operations, GPU vs CPU, automatic differentiation (autograd)
• Neural networks: artificial neurons, perceptrons, multi-layer perceptrons (MLPs), activation functions (ReLU, sigmoid, softmax)
• Training your first model: forward propagation, loss functions (cross entropy), backpropagation, gradient descent, optimizers (SGD, Adam)
• Model evaluation: accuracy, precision, recall, confusion matrix, overfitting vs underfitting

Session activity

Train a CNN for multiclass image classification.

Learning objectives

• Learn how convolutional neural networks extract visual features for image classification.

Key concepts introduced

• Why CNNs: limitations of MLPs, spatial information
• CNN building blocks: convolution, kernels, feature maps, padding, stride, pooling (max, average)
• CNN architecture: input layer, convolution layers, activation functions, pooling layers, fully connected layers, softmax output
• Training CNNs: data augmentation, batch normalization, dropout, hyperparameter tuning
• Evaluation: learning curves, confusion matrix, precision and recall

Session activity

Fine-tune ResNet-50 on a custom dataset that the class prepares and annotates in CVAT.

Learning objectives

• Fine-tune pre-trained models and prepare high-quality datasets.

Key concepts introduced

• Transfer learning: what it is, feature extraction, fine-tuning, ResNet-50, residual connections, model architecture (high level), loading pretrained weights, freezing and unfreezing layers
• Dataset engineering: collecting images, dataset organization, train/validation/test split, data augmentation, handling class imbalance
• Dataset annotation: introduction to CVAT, creating projects, image annotation, exporting annotations
• Training best practices: learning rate scheduling, early stopping, model checkpointing

Session activity

Train a custom YOLO detector and run real-time object detection.

Learning objectives

• Detect and localize multiple objects in real time.

Key concepts introduced

• Introduction to object detection: classification vs localization vs detection, bounding boxes, confidence scores, intersection over union (IoU)
• YOLO architecture, dataset structure, annotation formats, training pipeline
• Custom object detection: preparing a custom dataset, training on custom classes, validation, inference, real-time webcam detection
• Model evaluation: precision, recall, mAP, confusion matrix

Session activity

Build a real-time face recognition system using RetinaFace and ArcFace.

Learning objectives

• Detect, identify and verify faces using modern computer vision models.

Key concepts introduced

• Face detection: introduction, RetinaFace architecture (high level), facial landmarks, face alignment
• Face recognition: face embeddings, ArcFace architecture (high level), similarity metrics (cosine similarity, Euclidean distance)
• Recognition pipeline: face detection, face alignment, embedding extraction, identity matching
• Building a face database: enrollment, verification, identification
• Ethical considerations: privacy, bias, responsible AI

Session activity

Work on the capstone, combining the classification, detection and recognition capabilities built across Weeks 3 to 7.

Learning objectives

• Integrate multiple computer vision techniques into a complete application.

Key concepts introduced

• Capstone project: integrate the techniques learned throughout the course into a complete computer vision application
• Complete the capstone project