Towards Explainable, Safe Autonomous Driving with Language Embeddings for Novelty Identification and Active Learning: Framework and Experimental Analysis with Real-World Data Sets

This research explores the integration of language embeddings for activelearning in autonomous driving datasets, with a focus on novelty detection.Novelty arises from unexpected scenarios that autonomous vehicles struggle tonavigate, necessitating higher-level reasoning abilities. Our proposed methodemploys language-based representations to identify novel scenes, emphasizingthe dual purpose of safety takeover responses and active learning. The researchpresents a clustering experiment using Contrastive Language-Image Pretrained(CLIP) embeddings to organize datasets and detect novelties. We find that theproposed algorithm effectively isolates novel scenes from a collection ofsubsets derived from two real-world driving datasets, one vehicle-mounted andone infrastructure-mounted. From the generated clusters, we further presentmethods for generating textual explanations of elements which differentiatescenes classified as novel from other scenes in the data pool, presentingqualitative examples from the clustered results. Our results demonstrate theeffectiveness of language-driven embeddings in identifying novel elements andgenerating explanations of data, and we further discuss potential applicationsin safe takeovers, data curation, and multi-task active learning.

Further reading#

• Access Paper in arXiv.org