Beyond Count Balancing: Using Vision-Language Models to Build Visually Diverse Training Datasets

When building an image classifier, the standard advice is: "balance your classes." Make sure each class has roughly the same number of images. But for fine-grained classification — identifying 3,800+ plant species — equal counts per class is the easy part. The hard part is within-class visual diversity.

Consider Kalanchoe blossfeldiana (Flaming Katy). Our dataset had 249 images — a perfectly healthy count. But when we looked closer:

The model was learning "Flaming Katy = red flower." A user photographing a yellow or white cultivar? Misclassified.

This wasn't an isolated case. Across our 3,819 species:

• 48 classes were >80% flower images — the model couldn't recognize the plant without flowers
• 199 classes were >80% leaf images — the model failed when the plant was flowering
• Only 30 classes had balanced flower/leaf representation

Count balancing was a solved problem. Visual diversity balancing was the real gap.

---

The Insight: Your Embeddings Already Know

The breakthrough was realizing that a vision-language model like SigLIP already understands the visual axes we care about — flowers vs. leaves, color variants, growth stages — we just need to ask it.

SigLIP (Sigmoid Loss for Language Image Pre-Training) maps images and text into the same embedding space. This gives us two powerful capabilities:

1. Zero-shot classification: Compare image embeddings against text prompts like "a red flower" or "green leaves only" to categorize without any labeled data
2. Semantic clustering: Images that look similar (same color, same composition) cluster together in embedding space

We embedded all 788,467 images using ViT-B-16-SigLIP-256 and stored them in a SQLite cache (resumable — important when processing takes hours on consumer hardware).

---

Three Axes of Visual Diversity

Axis 1: Flower vs. Vegetative vs. Fruiting

We created text embedding vectors for each category and computed similarity scores:

flower_vec = mean_embed(["a photo of a plant with flowers blooming",
                         "a close-up photo of a single flower"])

leaf_vec = mean_embed(["a photo of a plant with leaves only, no flowers",
                       "a close-up photo of a leaf"])

relative_score = image_embed @ flower_vec - image_embed @ leaf_vec

A positive score means "flower-like," negative means "leaf-like." A threshold of 0.03 separated the three groups cleanly.

Axis 2: Flower Color

Same approach, different text prompts:

color_prompts = ["a red flower", "a pink flower", "a orange flower",
                 "a yellow flower", "a white flower", "a purple flower",
                 "green leaves only"]

color_label = argmax(image_embed @ color_vecs.T)

This gave us a color distribution per species — without any manual labeling.

Axis 3: Visual Composition (via Clustering)

For variation that text prompts can't capture — close-up vs. whole plant, indoor vs. outdoor, different backgrounds — we used KMeans clustering on the raw embeddings. We automatically selected the best k (2-8) using silhouette score.

---

The Sampling Strategy

With these three axes, we built a hierarchical sampling pipeline:

For each species (cap = 300 images):
  1. Classify all images as flower / leaf / ambiguous
  2. Reserve 15% minimum for the minority type (flower or leaf)
  3. Within the flower pool:
     a. Group by color (red, pink, orange, yellow, white, purple)
     b. Each present color gets at least 8% floor
     c. Remaining quota distributed proportionally
     d. Within each color group, use KMeans clustering for diversity
  4. Within the leaf pool:
     Use KMeans clustering for compositional diversity
  5. Resize all to 256px short side, JPEG q=85

The 15% minority floor means that even if a species has 92% flower images (like Zinnia elegans), the balanced dataset will include at least 40+ leaf images — enough for the model to recognize the plant without flowers.

The 8% color floor means that if Portulaca grandiflora has 5 color variants in the data, each gets at least ~5 images, even if 70% of the source data is pink.

---

What We Found

The Bimodality Problem

SigLIP clustering revealed classes with two visually distinct populations (high silhouette score at k=2). Some were legitimate diversity (flower vs. leaf). Others were data contamination — wrong species mixed in.

The Synonym Problem

Cross-class centroid similarity revealed 11 species pairs that were taxonomic synonyms — the same plant under different scientific names:

These weren't just similar-looking species — they were identical embeddings because the training images were literally the same plant. Without SigLIP analysis, we wouldn't have caught these.

The Color Dominance Problem

Of the 48 flower-heavy species, most were dominated by a single color variant. The balancing pipeline boosted minority colors:

Kalanchoe blossfeldiana — Before vs. After (cap=270 train):

• Red: 68% → ~45%
• Orange: 7% → ~15%
• Yellow: 5% → ~12%
• Pink: 4% → ~10%
• Leaves: 15% → 15% (preserved by minority floor)

---

The Numbers

The dataset went from 47 GB to ~8 GB — a 6x reduction — while actually improving visual coverage within each class.

---

What Balancing Can't Fix

This approach maximizes diversity of what exists. It can't create images that aren't in the source data. If your dataset has zero white Kalanchoe images, no balancing strategy will fix that.

The solution is a gap analysis pipeline:

1. Build a trait database (from GBIF, Wikidata, iNaturalist) that knows what color variants each species can have
2. Compare against the SigLIP color distribution of what we actually have
3. The difference is a sourcing list — specific species + color combinations to find on iNaturalist or other sources

Your external trait database is the ground truth for "what should exist." Your embeddings tell you "what does exist." The gap between them is your next data collection sprint.

---

Implementation Notes

Hardware: M-series Mac with MPS acceleration. Embedding 788K images took ~20 hours (bottlenecked by reading from a zip on an external USB drive — extracting first would be ~5x faster). Balancing + packing: ~30 minutes.

Model: ViT-B-16-SigLIP-256 via open_clip. 768-dimensional embeddings, stored as float16 in SQLite with WAL mode for crash-safe resumability.

Key libraries: openclip, scikit-learn (KMeans, silhouettescore), Pillow for resize, tqdm for sanity.

The cache is everything: Embedding is the expensive step. Store it in SQLite, make it resumable. The balancing strategy will change many times — re-embedding should never be necessary.

---

Takeaways

1. Count balance is necessary but not sufficient. Within-class visual diversity is the real differentiator for fine-grained classifiers.

1. Vision-language models are free diversity auditors. SigLIP gives you flower/leaf classification, color distribution, and contamination detection — all zero-shot, no labeling needed.

1. Cluster, don't random-sample. Random downsampling from 1,700 to 300 images will likely preserve the 68% red dominance. Stratified sampling across visual clusters preserves rare variants.

1. Know what you're missing. Balancing can only redistribute existing data. External trait databases tell you what variants should exist — that's your sourcing list.

1. Resize early. If training is at 224×224, storing 4000px originals is pure waste. Resize to 256px at dataset creation time. Our 47 GB dataset became 8 GB with zero information loss for training.

---

*This is part of our work building SASYA — a plant identification app focused on the Indian subcontinent's flora. The dataset covers 3,819 species with particular depth in tropical and Indian endemic plants.*