Choosing a Model for Plant Disease Detection: Datasets, Architectures, and Training
A practical guide to plant disease detection: how to actually use a dataset like PlantVillage, how to choose a model architecture for your use case, and why fine-tuning an existing model usually beats training one from scratch.
Most guides to this jump straight to picking a model. The dataset is actually the harder, more important decision, and it constrains everything that comes after it.
Start with the dataset, not the model
The PlantVillage dataset (around 54,000 images, 14 crop species, 38 disease classes) is the usual starting point, and for good reason: it's public, labeled, and big enough to actually train on. But before touching a model architecture, a few things about the data matter more:
- Class balance. Some diseases have thousands of labeled examples, others a few hundred. A model trained on an imbalanced dataset looks accurate overall while quietly failing on the rare classes. Check the per-class counts before anything else.
- Background variance. PlantVillage photos are mostly plain-background, well-lit, single-leaf shots. A model trained only on that generalizes poorly to a real photo taken in an actual garden, with dirt, shadows, and other leaves in frame. If your real use case looks nothing like the training photos, the accuracy number from the paper won't hold up in practice.
- Split by plant, not by photo. If multiple photos of the same diseased leaf end up split across train and test, the model can effectively memorize that leaf, and the test accuracy will lie to you. Split by source plant or image, not by individual photo.
None of this is about the model yet. Get the data right first, or the model choice barely matters.
Choosing a model for your use case
Once the data is in order, the model choice mostly comes down to answering three questions honestly.
Where does inference run? On-device, offline, on a phone, means a small, fast architecture: MobileNetV2/V3, EfficientNet-Lite, or something similarly sized, usually quantized afterward. Model size and latency are real constraints, not just accuracy. Server-side means you can afford a bigger backbone, ResNet50, EfficientNet-B4 and up, or a vision transformer, since you're not shipping the weights to a phone or racing a camera framerate.
Live camera, or a single uploaded photo? A live-camera use case needs something fast enough to run many times a second, which usually means trading some accuracy for speed. This is the same detect-then-classify split I use for species ID: a small model runs continuously, a bigger one runs once, on a clean crop. A single-upload use case can afford a slower, more accurate model, since it only runs once.
One label, or several? Most disease datasets assume one label per photo. Real plants can have more than one problem at once, a nutrient deficiency and a fungal spot, say, which turns the model from a plain classifier into a multi-label problem: a different loss function, and a threshold per class instead of one softmax pick.
Existing model vs. training your own from scratch
This decision is easier than it sounds, and it almost always goes the same way: use an existing pretrained backbone, and fine-tune it, rather than training from scratch.
Transfer learning, starting from a model already trained on ImageNet or similar, then fine-tuning it on labeled disease photos, works well with a dataset the size of PlantVillage. Tens of thousands of images is enough, because the early layers already know general things like edges, textures, and color patterns. Fine-tuning only has to teach the disease-specific part.
Training from scratch needs a dataset an order of magnitude bigger to reach the same accuracy, since the model has to learn general visual features and the disease-specific ones at the same time. It's rarely worth it unless a dataset that large already exists, or the input isn't a normal RGB photo.
In practice: freeze the early layers of a pretrained backbone, replace the final classification layer with one sized for the disease classes at hand, and fine-tune the last few layers, plus the new head, on the labeled dataset. That's the approach behind most published plant disease detection results, and it's the one I'd start with too.
This is the same set of decisions a disease classifier for SASYA would go through, dataset first, then model, then training approach, whenever it gets built. Right now that's still the plan, not something shipped.