[Submitted on 16 Oct 2026]
Abstract:Large language models (LLMs) are increasingly used in annotation tasks with varying success. While several studies have demonstrated the opportunities and shortcomings of LLMs for annotation, a critical question remains unanswered: is it possible to predict how well they will perform on a new dataset? To answer this and related questions, we construct two task families which induce 461 text classification datasets. We find that it is possible to predict LLM performance – to a certain degree – and that contrary to other work which highlights the usefulness of LLM verbalizations of self-confidence, internal log-probabilities are the most predictive signal.
Submission history
From: Tong Lin [view email]
[v1] Fri, 16 Oct 2026 21:22:38 UTC (185 KB)
