Lab 5 · Species Classifier on MegaDetector Crops

Biological motivation

The "detect-then-classify" pipeline is the standard production approach for camera traps. A generic detector (MegaDetector) finds animals; a project-specific classifier identifies them. The advantage: you can swap the classifier when species change without re-training detection. In this lab you train the classifier half on Oregon Critters crops.

Data

Cropped JPGs of single animals (one animal per image), 8 classes:

oregon_critters_crops/
├── train/
│   ├── deer/        ~600
│   ├── black_bear/  ~300
│   ├── cougar/      ~80
│   ├── coyote/      ~200
│   ├── elk/         ~250
│   ├── bobcat/      ~120
│   ├── raccoon/     ~180
│   └── other/       ~150
└── val/             (~25% per class, by site)

Tasks

Task 1 — fine-tune ResNet-50 (5 pts)

Adapt lab2_train.py to:

• Use torchvision.models.resnet50 with ImageNet weights.
• Replace the final layer for 8 classes.
• Use ImageFolder instead of a custom Dataset.
• Train for 15 epochs with random horizontal flips, mild color jitter, and RandomResizedCrop(224, scale=(0.8, 1.0)).

Task 2 — honest evaluation (5 pts)

Report:

• Overall top-1 accuracy on val.
• Per-class precision, recall, F1 (use sklearn.metrics.classification_report).
• An 8x8 confusion matrix as a heatmap (confusion.png).
• The 5 worst-confused class pairs.

Task 3 — calibration (5 pts)

A model that's right 70% of the time might say 0.99 on most of its outputs. That's a calibration problem and it matters for downstream occupancy modeling (Lab 10).

Compute a reliability diagram: bin val predictions by predicted confidence (10 bins from 0–1), then plot the fraction actually correct in each bin vs the bin center. Save as reliability.png. A perfectly calibrated model lies on the diagonal.

Task 4 — written report (5 pts)

In 250 words: discuss the worst-confused pairs (often biologically reasonable — coyote vs. small dog?), the calibration plot, and what your number means for someone using this classifier on a 10,000-image deployment.

Go-further

Add a "unknown" output: at inference, if the top class probability is below 0.6, label as unknown. How does this change precision/recall per species, and what fraction of images become unknown?

Rubric