Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factors

Test-time adaptation (TTA) fine-tunes pre-trained deep neural networks forunseen test data. The primary challenge of TTA is limited access to the entiretest dataset during online updates, causing error accumulation. To mitigate it,TTA methods have utilized the model output’s entropy as a confidence metricthat aims to determine which samples have a lower likelihood of causing error.Through experimental studies, however, we observed the unreliability of entropyas a confidence metric for TTA under biased scenarios and theoreticallyrevealed that it stems from the neglect of the influence of latent disentangledfactors of data on predictions. Building upon these findings, we introduce anovel TTA method named Destroy Your Object (DeYO), which leverages a newlyproposed confidence metric named Pseudo-Label Probability Difference (PLPD).PLPD quantifies the influence of the shape of an object on prediction bymeasuring the difference between predictions before and after applying anobject-destructive transformation. DeYO consists of sample selection and sampleweighting, which employ entropy and PLPD concurrently. For robust adaptation,DeYO prioritizes samples that dominantly incorporate shape information whenmaking predictions. Our extensive experiments demonstrate the consistentsuperiority of DeYO over baseline methods across various scenarios, includingbiased and wild. Project page is publicly available athttps://whitesnowdrop.github.io/DeYO/.

Further reading#

• Access Paper in arXiv.org