KD transfers the learned representations of a large teacher model into a smaller student model. Since the original KL-divergence formulation, ¹ which trains the student to match the teacher's softened output distribution. However, a wave of methods has moved beyond matching pointwise logits to care about structural representations instead. Relational methods like RKD ² penalize differences in pairwise distance between samples. Contrastive methods like CRD ³ and RRD ⁴ maximize mutual information between teacher and student representations using negative samples. These methods consistently outperform pointwise approaches which hinted to us that the geometric structure in the latent space is a stronger signal for distillation than raw activation matching. It is noteworthy that these methods treat every pair equally. The contrastive loss spends the same gradient on sample pairs the student has already captured and pairs where the student is completely lost. We hypothesized that representational interpretability methods could diagnose where the student is wrong and direct the distillation signal there, yielding better performance from the same training budget.

#1.2 Representational Difference Explanations

Kondapaneni et al introduced RDX to effectively explain contrasting model representations. ⁵ Given two embedding matrices over the same data, RDX computes rank-normalized pairwise distances and applies a locally-biased difference function that amplifies disagreements between nearby samples while suppressing those between distant ones. The result is an affinity matrix that reveals groups of samples one model considers similar but the other does not. We repurpose its core machinery (rank normalization + locally-biased difference) as a supervision signal inside a distillation loss. Instead of producing post-hoc explanations, we use RDX to compute per-sample confusion weights and to select contrastive pairs, both refreshed every

R

epochs during training. To our knowledge, this is the first application of representational difference analysis as a direct component of a distillation objective.

RDX applied to MNIST digits 3, 5, and 8, revealing clusters that one model groups tightly while the other separates them. — Figure 2: RDX-guided selection on MNIST digits 3, 5, 8 with a student model $M_S$ and expert model $M_E$ . We reproduce the MNIST- $\{3,5,8\}$ comparison from the RDX paper using the official RDX code, and annotate the projections to show how RDX informs positive and negative selection. Left: $RDX(M_S ,M_E)$ reveals clusters unique to the student: samples the student groups tightly (cyan cluster, top-left PCA) that the expert separates (top-right PCA). These are the hard negatives: the student wrongly merges 3s, 5s, and 8s that the teacher distinguishes, identified by high $A_{ij}$ in the affinity matrix. Right: $RDX(M_E, M_S)$ reveals structure unique to the teacher: samples the expert clusters coherently that the student scatters. These inform positive selection: teacher-close pairs the student has yet to learn. Bottom grids show the corresponding image samples for each direction.

#2. Approach

We propose Representational Difference Distillation (RDD), a knowledge distillation framework that uses representational difference analysis to focus the distillation loss where the teacher and student structurally disagree.

#2.1 Identify Representational Misalignment

For every pair of samples

(i, j)

, compute ranks in teacher-space and student-space, then measure how much they disagree:

\text{diff}_{ij} = \tanh\!\left(\gamma \cdot \frac{r^T_{ij} - r^S_{ij}}{\min(r^S_{ij},\, r^T_{ij}) + 1}\right)

Positive values mean the student over-groups $i$ and $j$ ; negative values mean the student has an alignment gap to close. This becomes two asymmetric affinity matrices with sharpness $\beta$ : $A^{01}$ (teacher-unique structure), $A^{10}$ (student-unique structure), and a per-sample confusion score $c_i$ . We use $\gamma = 0.1$ and $\beta = 0.8$ , and recompute everything over the full training set every $R = 20$ epochs.

#2.2 Guided Contrastive Loss

We build on CRD's memory-bank contrastive setup but make two changes ³. Positives are selected via $A^{01}$ , the strongest teacher-unique neighbor for each anchor. Negatives are drawn from a top- $K$ pool ranked by $A^{10}$ , samples the student wrongly merges. Each anchor's contrastive loss is then scaled by its normalized confusion score:

\mathcal{L}_{\text{RDD}} = \frac{1}{B}\sum_{i=1}^{B} \frac{c_i}{\bar{c}} \cdot \mathcal{L}^{\text{NCE}}_i

#2.3 Training Objective

The final training loss is:

\mathcal{L} = \lambda_{\text{CE}} \cdot \mathcal{L}_{\text{CE}} + \lambda_{\text{KD}} \cdot \mathcal{L}_{\text{KD}} + \lambda_{\text{RDD}} \cdot \mathcal{L}_{\text{RDD}}

Standard cross-entropy, standard KD, plus our RDX-weighted contrastive term. In the reported run $\lambda_{\text{KD}} = 0$ , so RDD is a pure feature-level method.

#3. Results

We evaluate on CIFAR-100 (50k train / 10k test, 100 classes) using the resnet32x4 → resnet8x4 teacher–student pair, following the protocol from CRD. All methods KD, RKD, CRD, and ours have: SGD with momentum 0.9, weight decay

5 \times 10^{-4}

, initial learning rate 0.05 decayed by

0.1\times

at epochs 150, 180, and 210, for 240 total epochs at batch size 64. For the RDX diagnostic we set

\gamma = 0.1

\beta = 0.8

, and

R = 20

. RDD supervision uses

\lambda_{\text{CE}} = 1

\lambda_{\text{KD}} = 0

\lambda_{\text{RDD}} = 0.8

, the same weights as CRD, so the two methods differ only in the RDX-guided selection and weighting. The sampling and curriculum variants keep the KD objective (

\lambda_{\text{CE}} = 0.1

\lambda_{\text{KD}} = 0.9

) and use RDX only to choose which samples the student sees. Due to compute constraints, every number below is from a single run, so differences of a few tenths of a percent are within seed noise. The full hyperparameters and commands for each run are in the repo README.

#3.1 Naive Sampling

Our first instinct was to use RDX for data selection: rank the training set by RDX difficulty and train only on a subset. There is good precedent for active sampling methods like RHO Loss, ⁶ and we were initially optimistic. We trained on the easiest 15% of the training set by RDX ranking, recomputing the subset every $R$ epochs. It could not beat any distillation baseline. RDD sampling reached 59.49%, substantially below KD (72.93%), CRD (75.45%), and everything else. Without refreshing the ranking it was worse still, at 54.53%. The reason, in hindsight, is obvious: by restricting training to the easiest RDX-ranked samples, the student only ever sees the portion of the data where teacher and student already agree. It never encounters the challenging boundary cases that drive representation refinement. The student converges quickly to a solution that handles easy samples well but generalizes poorly.

sampling results on CIFAR-100. — Figure 3: CIFAR-100 resnet32x4→resnet8x4 test accuracy. RDD sampling trains on an RDX-ranked subset of 15% of the training set; the subset is recomputed every $R$ epochs. Curves shown for KD, CRD, RKD, and RDD sampling.

#3.2 Curriculum Learning

Instead of restricting to easy RDX-ranked samples, we start with them and gradually introduce harder ones through curriculum learning. ⁷ RDX confusion scores give us a principled difficulty ranking derived from actual structural disagreement. RDD curriculum reached 72.96%, matching KD (which uses the same loss), but falling short of CRD. The reason for this was that during the initial phase when only the easiest 30% of samples are available, the student learns a reasonable but incomplete representation. As harder samples come in, the student must reorganize its feature space to accommodate them, causing a visible slowdown around epochs 60-100. By epoch 150 all samples are active and performance catches up to KD.

#3.3 RDX-Weighted Supervision

Our final method, RDX-weighted supervision, reaches 75.58% test accuracy. That is on par with CRD (75.45%), and clearly above KD (72.93%) and RKD (71.55%), as shown in Figure 5. Because RDD and CRD share the same loss weights, projection heads and memory bank, this comparison isolates the effect of RDX-guided selection and weighting. A possible explanation for the late-training behavior is the asymmetric use of the two RDX affinity matrices. While CRD samples negatives uniformly, RDD draws them from the memory bank entries that

A^{10}

identifies as wrongly merged by the student, so more of its gradient budget goes to the pairs where the student's neighborhood structure actually deviates from the teacher's.

Method	Test Acc. (%)
KD (Hinton et al., 2015)	72.93
CRD (Tian et al., 2020)	75.45
RKD (Park et al., 2019)	71.55
RDD Sampling (ours)	59.49
RDD Curriculum (ours)	72.96
RDD Supervision (ours)	75.58

Figure 5: Final-epoch test accuracy (%) for resnet32x4→resnet8x4 on CIFAR-100, single run per method. The teacher (resnet32x4) reaches 79.42%.

RDD table supervision. — *Figure 6:* CIFAR-100 resnet32x4→resnet8x4 test accuracy. RDD supervision uses multi-negative contrastive distillation weighted by per-sample RDX affinity to emphasize teacher-student representational disagreement. Curves shown for KD, CRD, RKD, and RDD supervision.

#4. Discussion

- Weighting beats filtering, in our experiments: If you have a signal about sample difficulty, use it to modulate your loss rather than to exclude data. (The curriculum and supervision variants also differ in their loss, KD versus contrastive, so this comparison doesn't fully isolate data selection.)

- RDX staleness is a real problem. We refresh the affinity matrices every $R = 20$ epochs, but the student's representation evolves between refreshes. In early training when the student is changing fast, the affinities go stale quickly. We suspect an adaptive refresh schedule, triggered by shifts in the confusion distribution rather than a fixed clock, would help. We didn't solve this.

- The margin over CRD is narrow. 75.58% vs 75.45% is a single run each and within seed noise, so we consider them on par. The gap does appear only late in training, where targeted weighting should matter most, but multi-seed runs are needed to confirm it. Whether this holds for cross-architecture transfers or at ImageNet scale remains open.

- The negative result is the most informative result. The 59.49% sampling collapse taught us more about distillation dynamics than the 75.58% supervision result. If your method can't survive data restriction, it's telling you something fundamental about the role of diversity in representation learning.

#5. Acknowledgments

I want to thank Abdulkarim Mugisha for working with me on this project. I would also like to thank Prof. Pietro Perona, for his guidance on experiments and paper writing, and Neehar Kondapaneni for his guidance and mentorship.

#6. References

[1]
Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network, 2015. URL https://arxiv.org/abs/1503.02531.
[2]
Park, W., Kim, D., Lu, Y., and Cho, M. Relational knowledge distillation. In CVPR, 2019. URL https://arxiv.org/abs/1904.05068.
[3]
Tian, Y., Krishnan, D., and Isola, P. Contrastive representation distillation. In ICLR, 2020. URL https://arxiv.org/abs/1910.10699.
[4]
Giakoumoglou, N. and Stathaki, T. Relational representation distillation, 2025. URL https://arxiv.org/abs/2407.12073.
[5]
Kondapaneni, N., Mac Aodha, O., and Perona, P. Representational difference explanations. In NeurIPS, 2025. URL https://arxiv.org/abs/2505.23917.
[6]
Mindermann, S., et al. Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt, ICML 2022.
[7]
Bengio, Y., Louradour, J., Collobert, R., and Weston, J. Curriculum learning. In Proceedings of the 26th International Conference on Machine Learning (ICML 2009), pp. 41–48, 2009.

#1. Background & Related Work