Curriculum Learning for Dental Disease Detection

A three-stage YOLOv8 pipeline on the DENTEX 2023 dataset, and an honest negative result.

DENTEX 2023 panoramic X-rays, YOLOv8m segmentation. A three-stage curriculum (quadrant → tooth → disease) versus a matched single-stage baseline: mAP@0.5 of 0.394 vs 0.417. The schedule didn’t help — class imbalance was the binding constraint.

Summary. A three-stage curriculum learning framework (quadrant localization, then tooth enumeration, then disease diagnosis) on the DENTEX 2023 panoramic X-ray dataset (2,032 hierarchically labeled images) using YOLOv8m segmentation models. Against a matched single-stage baseline, the curriculum approach achieved mAP@0.5 of 0.394 versus 0.417, a small but real regression. The empirical takeaway is that on this size of dataset, additional weakly-related supervision didn’t help fine-grained detection. Class imbalance was the dominant limitation, not the training schedule.

Note

This was my final project for DSAN 6600, Neural Networks & Advanced Deep Learning at Georgetown (Spring 2026).

The question

Curriculum learning, training models on easier sub-tasks before harder ones, has a strong intuitive appeal, especially for hierarchical labels. Dental panoramic X-rays are a near-perfect test bed. Every tooth lives in a quadrant, has a number, and may or may not have one of several conditions. Does staging the supervision in that order actually help fine-grained disease detection on a small medical dataset?

Approach

Working with a teammate (Tianyu Zhao), we built on the DENTEX 2023 dataset (Hugging Face, ibrahimhamamci/DENTEX), which ships with hierarchical annotations at three tiers: quadrant localization, quadrant + tooth enumeration, and full disease diagnosis with masks. I led the curriculum-learning half of the project; my teammate led two parallel questions on image-degradation robustness and region-focused preprocessing.

The curriculum model trains a YOLOv8m segmentation network across three sequential stages, quadrant localization (30 epochs), tooth enumeration (40 epochs), then disease diagnosis (60 epochs), with each stage initialized from the previous stage’s weights. That’s compared against a single-stage YOLOv8m baseline trained directly on the Stage 3 disease-detection data, matched on architecture and total training budget. The four disease classes in the final task (impacted tooth, caries, periapical lesion, deep caries) are heavily imbalanced, which turns out to be the dominant factor in the results below.

# Sketch of the curriculum schedule. Full code in the repo.
stages = [
    {"task": "quadrant_localization", "epochs": 30, "data": "quadrant_labels"},
    {"task": "tooth_enumeration",     "epochs": 40, "data": "tooth_labels"},
    {"task": "disease_diagnosis",     "epochs": 60, "data": "disease_labels"},
]

Results

Models were evaluated on mAP@0.5 and mAP@0.5:0.95, with the best checkpoint per run selected on the validation set and reported on held-out test data. The headline number: the three-stage curriculum model scored mAP@0.5 of 0.394, against 0.417 for the matched single-stage baseline. The curriculum didn’t just fail to help, it introduced a small but consistent regression.

The rest of the project (led by my teammate) puts that result in context. A separate robustness experiment found that a clean-trained YOLOv8m model held up reasonably well under blur and motion blur but degraded sharply under Gaussian noise, and that training on degradation-augmented data recovered most of that loss. A third experiment found that cropping to the dental region of interest, or to individual quadrants, measurably improved disease detection over using the full panoramic image. Read together, the pattern is that geometric supervision (curriculum stages, cropping) helps when it directly narrows the search space, but doesn’t substitute for addressing the underlying class imbalance in the disease labels.

What I learned

The interesting part of this project wasn’t the architecture. It was sitting with a result that didn’t go the way I expected and figuring out why. Two things stood out.

1. The class distribution was doing more work than the schedule. A small handful of disease classes dominated. A curriculum that doesn’t address that imbalance just front-loads the easy stages without solving the actual problem.
2. “More supervision” is not a free lunch on small datasets. Each curriculum stage adds variance from its own labels. If those labels are only weakly related to the downstream task, you can pay the variance cost without earning the bias reduction.

What I’d do differently

Given what the rest of the project turned up, the next version wouldn’t be “more curriculum,” it would be curriculum plus the two things that actually moved the needle elsewhere in this project: region-focused cropping (train the disease head on ROI or quadrant crops, not the full panoramic image) and an explicit fix for class imbalance, focal loss or class-rebalanced sampling, rather than hoping staged supervision would absorb it implicitly. I’d also want to ablate the curriculum stages individually to see whether any single stage (versus the full three-stage schedule) contributes anything at all.

Code

Repository on GitHub