Can a Generalist Model Learn to Specialize? Three Experiments with SAM

Wood Texture Dataset Structure

wood_data/
├── images/
│   └── images/
│       ├── abura-s-60x60.jpg
│       ├── african-blackwood-60x60.jpg
│       ├── oak-60x60.jpg
│       ├── walnut-60x60.jpg
│       └── ... (530 total images)
└── wood.csv (metadata with species names)

Each image is a 60×60 or similar resolution texture sample. We created 500 synthetic 512×512 mosaics (2×2 grids of 256×256 tiles) for training.

The Setup

Dataset: 530 wood texture images from various species (oak, walnut, pine, etc.)
Model: MobileSAM (the lightweight variant with a TinyViT image encoder)
Training Data: 500 synthetic 2×2 mosaics (each containing 4 different wood textures)
The Goal: Teach SAM to distinguish between visually similar wood grains We started by creating “mosaic” images, think of them as 2×2 grids where each quadrant contains a different wood texture. To make things interesting, we used ResNet18 to extract features from all 530 images, then applied K-Means clustering to group similar woods together. This gave us 5 distinct clusters:

Cluster 1: 183 images (34.5%) - Lighter hardwoods
Cluster 0: 171 images (32.3%) - Medium tones
Cluster 2:  96 images (18.1%) - Darker woods
Cluster 4:  41 images (7.7%)  - Exotic species
Cluster 3:  39 images (7.4%)  - Very distinct grains

Baseline Evaluation (Zero-Shot Mobile Sam)

We gave MobileSAM the perfect center point of each tile. The thought here was:

• If it's a good generalist, it should segment the square tile perfectly.
• If it fails, it will likely over-segment (texture details) or under-segment (merge similar woods).

Figure: Zero shot (Average Baseline IoU for this image: 0.9950)

The “Hard Mode” Challenge

Here’s where it got interesting. We created two test modes:
- Standard Mode: Mosaics with textures from different clusters ( lots of visual contrast)
- Hard Mode: Mosaics where ALL four tiles came from the SAME cluster ( similar colors, only grain direction differs) The hypothesis was simple: if our specialized model could learn fine-grained texture boundaries, it should outperform vanilla SAM on Hard Mode.

The Training

We ran a straightforward training loop:
- 5 epochs
- Learning rate: 1×10⁻⁴
- Loss function: Dice loss only
- Prompt strategy: Perfect center point of each tile The training seemed to go great. The loss dropped beautifully:

Epoch 1: Loss = 0.1570
Epoch 2: Loss = 0.0125  (92% drop!)
Epoch 3: Loss = 0.0426  (wait, it went UP?)
Epoch 4: Loss = 0.0002  (okay, back down...)
Epoch 5: Loss = 0.0001  (basically zero!)

We were thrilled. Loss of 0.0001? That’s incredible! Time to test it.

The Devastating Results

Base MobileSAM mostly keeps to the tile edges, while the specialized model's masks spill across tile boundaries or cover only part of the tile.

The decrease was 36.28 percentage points of mIoU (about 37.9% relative to the baseline). On the one standard mosaic we inspected, vanilla MobileSAM averaged 99.5% IoU across its four tiles. But on Hard Mode (similar textures), base SAM scored 95.8% while our “specialized” version completely collapsed to 59.5%.

What the failed evaluation told us

The training loss fell to 0.0001 while held-out performance fell. That is consistent with overfitting. We had no validation split, and the small synthetic training set limited what we could conclude. This run did not isolate the effects of dataset size, loss, or learning rate.

Challenge 2: Carpet Defect Detection - The Synthetic Data Experiment

The Setup

Dataset: MVTec AD Carpet Dataset
Model: MobileSAM (same as wood experiment)
Real Training Data: 280 defect-free “good” carpet images
Real Test Data: 89 images with real industrial defects (5 types: cuts, holes, thread pulls, metal contamination, color defects)
The Problem: 89 defect images is way too few to train a neural network This was an industrial anomaly detection challenge. Imagine a carpet factory where cameras scan every meter of output looking for manufacturing flaws. These defects are subtle, random, and expensive to collect labeled examples of.

Carpet Defect Dataset Structure

carpet/
├── train/
│   └── good/ (280 defect-free images)
├── test/ (89 defect images across five classes)
│   ├── color/
│   ├── cut/
│   ├── hole/
│   ├── metal_contamination/
│   └── thread/
└── ground_truth/
    ├── color/ (*_mask.png files)
    ├── cut/
    ├── hole/
    ├── metal_contamination/
    └── thread/

Total: 280 training images (all good), 89 test images with defects. We generated 500 synthetic defect images from the good images for training.

The Synthetic Data Solution

Since we couldn’t train on real defects (too few), we built a synthetic defect generator that could inject realistic-looking anomalies into the 280 good images: Type 1: Patches (simulates holes, metal contamination, cuts)
- Cut out random 20-60px patches from the carpet
- Paste them elsewhere, but darkened (40-70% brightness) or with added Gaussian noise
- Creates the appearance of holes or foreign objects Type 2: Scars (simulates thread pulls, cuts)
- Draw thin random lines (2-5px thick) across the carpet
- Use dark colors (0-50 RGB values) to simulate broken threads
- Random start/end points to create irregular patterns Type 3: Stains (simulates color defects)
- Apply circular color shifts (40-100px radius) using HSV manipulation
- Shift the hue by 20-50 degrees
- Blend at 40% opacity for a realistic “dye bleeding” effect We generated 500 synthetic training samples this way, each one with a known mask and a center-point prompt at the anomaly location.

Three synthetic training samples, each with a red star marking the center-point prompt at the generated defect.

The Training

Learning from the wood disaster, we kept a close eye on the training dynamics:

Epoch 1: Loss = 0.2544
Epoch 2: Loss = 0.2006  (21% drop - healthy)
Epoch 3: Loss = 0.1818  (9% drop)
Epoch 4: Loss = 0.1639  (10% drop)
Epoch 5: Loss = 0.1527  (7% drop)

The curve was smoother than the wood run. We needed the real-defect evaluation to assess generalization; training loss alone could not establish it.

Training config:
- 5 epochs
- Learning rate: 1×10⁻⁴
- Loss: Dice only
- 500 synthetic samples, tested on 89 REAL defects

The Results: Modest but Real Improvement

Not spectacular, but positive! We improved by 6.75 percentage points (a 23% relative improvement) without the model ever seeing a single real defect during training.

These are some outputs from the Base Model: Fig: Visual comparison showing Baseline SAM

These are some of the outputs from our specialized model: Fig: Visual comparison showing Specialized SAM(Our model)

What this result supports

The real-defect evaluation supports a modest transfer from synthetic training to real images. The synthetic patches, scars, and stains are simpler than real manufacturing defects, so this was a starting point for further evaluation.

Challenge 3: Camouflaged Animal Detection

The setup

We used SAM ViT-H with its encoder frozen and adapted the mask decoder. The saved class-project record describes a COD10K-v3 copy with 6,000 training files and 4,000 test files, plus horizontal-flip augmentation. Those are file counts from that record, not counts of camouflaged animals. The headline evaluation used 200 sampled test images; the eligible subset and random seed would need the original manifest to reproduce.

Camouflage Dataset (COD10K-v3)

COD10K-v3/
├── Train/
│   ├── Image/ (6,000 images)
│   │   ├── COD10K-CAM-1-Aquatic-1-BatFish-1.jpg
│   │   ├── COD10K-CAM-1-Aquatic-2-Crab-10.jpg
│   │   └── ...
│   └── GT_Object/ (6,000 binary masks)
└── Test/
    ├── Image/ (4,000 images)
    └── GT_Object/ (4,000 masks)

The class-project notes record 12,000 samples after augmentation and evaluation on 200 selected test images.

The Dataset

COD10K contains images organized by categories:
- Aquatic animals (fish, octopuses, seahorses camouflaged against reefs)
- Terrestrial animals (insects, lizards, frogs blending into leaves/bark)
- Flying animals (moths with wing patterns matching tree bark) The camouflaged-object subset provides annotated masks; the full COD10K release also contains non-camouflaged and background images. The dataset is challenging because:
- Low contrast: Animal and background have similar colors
- Complex textures: Natural environments have high-frequency patterns everywhere
- Scale variation: Animals range from tiny insects to large reptiles
- Occlusion: Many animals are partially hidden

Figure: Showing sample: COD10K-CAM-1-Aquatic-1-BatFish-1.jpg

Training the decoder

We froze the image encoder and cached embeddings. The camouflage run used a combined pixel-level and overlap loss, with point and box prompts sampled in equal proportions.

# Pseudocode for the combined objective
bce_loss = Binary_Cross_Entropy(pred, target)
dice_loss = 1 - Dice_Coefficient(pred, target)
final_loss = 0.5 * bce_loss + 0.5 * dice_loss

The Training

The recorded losses across seven epochs:

Epoch 1: Loss = 0.1420
Epoch 2: Loss = 0.1272  (10.4% drop)
Epoch 3: Loss = 0.1172  (7.9% drop)
Epoch 4: Loss = 0.1140  (2.7% drop)
Epoch 5: Loss = 0.1113  (2.4% drop)
Epoch 6: Loss = 0.1081  (2.9% drop)
Epoch 7: Loss = 0.1059  (2.0% drop)

Total: 25.4% reduction, NO COLLAPSE

The final training loss was 0.1059.

The Breakthrough Results

The headline class-project summary records an 18.83 percentage-point gain, about 39.6% relative to its baseline. The retained comparisons show both improvements and cases where the models behaved similarly. Benchmark comparisons: input, baseline mask, and specialized mask. These examples illustrate behavior; the aggregate metrics are reported separately.

Limits of the comparison

The runs changed several factors together. They do not establish which change caused the gain, and no random-mask baseline was measured. The prompt comparison below is a separate result table, with its own values.

Real-World Test: The Smithsonian Zoo

We didn’t just test on the COD10K dataset; we took our model to the Smithsonian National Zoo and photographed real animals in their enclosures to see how it performed on photos outside our benchmark evaluation.

Figure: Images from our visit to the Zoo

Additional examples retained from the project notes and attributed there to Google. They are qualitative examples, separate from the zoo photos and benchmark scores.

Multi-point comparison

The Comparative Analysis

These are the recorded class-project results. They are not a controlled comparison across tasks.

Performance Summary