[Submitted on 27 Feb 2026 (v1), last revised 23 Oct 2026 (this version, v2)]
Abstract:When a text description is extended with an additional detail, image-text similarity should drop if that detail is wrong. We show that CLIP-style dual encoders often violate this intuition: appending a plausible but incorrect object or relation to an otherwise correct description can increase the similarity score. We call such cases half-truths. On COCO, CLIP prefers the correct shorter description only 34.0% of the time, and performance drops to 27.8% when the added detail is a relation. We trace this vulnerability to weak supervision on caption parts: contrastive training aligns full sentences but does not explicitly enforce that individual entities and relations are grounded. We propose CS-CLIP (Component-Supervised CLIP), which decomposes captions into entity and relation units, constructs a minimally edited foil for each unit, and fine-tunes the model to score the correct unit above its foil while preserving standard dual-encoder inference. CS-CLIP raises half-truth accuracy to 76.4% and improves average performance on established compositional benchmarks by 5.8 points, suggesting that reducing half-truth errors aligns with broader gains in compositional understanding.
Submission history
From: Bora Kargi [view email]
[v1] Fri, 27 Feb 2026 10:56:31 UTC (4,916 KB)
[v2] Fri, 23 Oct 2026 13:29:55 UTC (3,247 KB)
