[Submitted on 30 Oct 2024 (v1), last revised 18 Nov 2026 (this version, v5)]
Abstract:State-of-the-art Extreme Multi-Label Text Classification models rely on multi-label attention to focus on key tokens in input text, but learning good attention weights is challenging. We introduce PLANT (Pretrained and Leveraged AtteNTion), a plug-and-play strategy for initializing the label-attention module that operates on top of an LLM’s output representations. PLANT works by planting label-specific attention using a Learning-to-Rank model trained on mutual information gain, before end-to-end fine-tuning. This architecture-agnostic approach integrates with LLM backbones (e.g., Mistral-7B, Llama-3.1-8B, DeepSeek-V3, and Phi-3-mini). PLANT outperforms SOTA methods across tasks like ICD coding, legal topic classification, and content recommendation, and improves fine-tuned LLMs in controlled comparisons outside the medical domain. Gains are largest on rare labels, although a training-free LLM extraction baseline remains competitive on rarely seen labels. Ablation studies confirm that attention initialization is a key driver of these gains. We make our code available at this https URL
Submission history
From: Debjyoti Saha Roy [view email]
[v1] Wed, 30 Oct 2024 14:41:23 UTC (8,495 KB)
[v2] Sun, 5 Oct 2025 05:55:06 UTC (417 KB)
[v3] Tue, 23 Dec 2025 03:28:14 UTC (469 KB)
[v4] Fri, 26 Dec 2025 01:31:59 UTC (443 KB)
[v5] Wed, 18 Nov 2026 04:57:37 UTC (716 KB)
