Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models

We introduce Groma, a Multimodal Large Language Model (MLLM) with groundedand fine-grained visual perception ability. Beyond holistic imageunderstanding, Groma is adept at region-level tasks such as region captioningand visual grounding. Such capabilities are built upon a localized visualtokenization mechanism, where an image input is decomposed into regions ofinterest and subsequently encoded into region tokens. By integrating regiontokens into user instructions and model responses, we seamlessly enable Gromato understand user-specified region inputs and ground its textual output toimages. Besides, to enhance the grounded chat ability of Groma, we curate avisually grounded instruction dataset by leveraging the powerful GPT-4V andvisual prompting techniques. Compared with MLLMs that rely on the languagemodel or external module for localization, Groma consistently demonstratessuperior performances in standard referring and grounding benchmarks,highlighting the advantages of embedding localization into image tokenization.Project page: https://groma-mllm.github.io/.

Further reading#

• Access Paper in arXiv.org