Knowledge Distillation Using Entropy-Guided Dynamic Parameter Transfer for Wound Image Classification

The problem

Standard knowledge distillation compresses a large teacher model into a smaller student using a fixed temperature (T) and loss-balance weight (α) for every training sample. But not every sample is equally ambiguous — an easy, confidently-classified wound image and a genuinely hard, borderline one carry very different amounts of useful signal in the teacher's soft labels. Using one fixed T and α for both wastes the easy samples and under-uses the hard ones.

EGATS — Entropy-Guided Adaptive Temperature Scaling

Instead of a fixed temperature, EGATS scales it per sample based on the teacher's own prediction entropy: T = clamp(1.0 + β·H_scaled, 1.0, 5.0). Higher-entropy (more uncertain) predictions get a higher temperature, softening the teacher's output distribution exactly where the extra nuance matters most.

EGAAS — Entropy-Guided Adaptive Alpha Scaling

The same entropy signal drives α — the balance between the distillation loss and the standard cross-entropy loss — via a sigmoid: α = σ(2 − H). Confident, low-entropy samples lean on ground-truth labels; uncertain, high-entropy samples lean more on the teacher's softened distribution.

Contrastive feature alignment

On top of EGATS/EGAAS, an auxiliary cosine-similarity contrastive loss (weighted λ = 0.1) pulls the student's intermediate feature representations toward the teacher's, giving the student a second, feature-space signal beyond just matching output logits.

How it fits together

For every training image, the teacher's own uncertainty sets both knobs at once.

1. ResNet-34 teacherPredicts class probabilities for each training image
2. Entropy per sampleHow uncertain is the teacher about this image?
3. Adaptive T and αT from entropy (1 to 5); α = σ(2 − H)
4. Distillation lossα·KL at temperature T + (1 − α)·cross-entropy + 0.1·alignment
5. ResNet-18 student42.7 MB, 49% smaller than the teacher

Dataset & setup

Teacher: ResNet-34, 20 epochs, LR 0.0015. Student: ResNet-18, 70 epochs, LR 0.001. Trained and evaluated on a Kaggle wound-image dataset spanning 9 classes — burns, bruises, cuts, venous wounds, abrasions, surgical wounds, pressure wounds, diabetic wounds, lacerations — balanced to 100 images per class (900 total), resized to 224×224, augmented with 9 Albumentations transform types, split 80/10/10.

Code and reproducibility

The notebook, the entropy-guided loss, a manifest of the 900-image subset and the five trained models (release assets with SHA-256 checksums) are public on GitHub, together with notes on the limits of the evaluation.

Through RAC

Produced through IPST/PMU-B's national RAC (Robotics, AI, and Coding) program, Cohort 2 — selected as 1 of 25 students nationally from a Python coding assessment, then developed and researched the method over an intensive residential workshop and follow-up remote work before submission to ICCI 2025.