DivLogicEval: A Framework for Benchmarking Logical Reasoning Evaluation in Large Language Models

Abstract

Logic reasoning in natural language has been recognized as an important measure of human intelligence for Large Language Models (LLMs). Popular benchmarks may entangle multiple reasoning skills and thus provide unfaithful evaluations on the logic reasoning skill. Meanwhile, existing logic reasoning benchmarks are limited in language diversity and their distributions deviate from the distribution of an ideal logic reasoning benchmark, which may lead to biased evaluation results. This paper thereby proposes a new classical logic benchmark DivLogicEval, consisting of natural sentences composed of diverse statements in a counterintuitive way. To ensure a more reliable evaluation, we also introduce a new evaluation metric that mitigates the influence of bias and randomness inherent in LLMs. Through experiments, we demonstrate the extent to which logical reasoning is required to answer the questions in DivLogicEval and compare the performance of different LLMs in conducting logical reasoning.

Why a New Logic Benchmark?

Entangled skills overestimate ability. Popular benchmarks (ReClor, LogiQA) mix commonsense and other reasoning with logic, so they can overestimate pure logical-reasoning ability. Prompting a model to avoid logical reasoning can paradoxically improve its score on them, but not on DivLogicEval.
Counterintuitive, diverse statements. DivLogicEval composes natural sentences from SNLI/MNLI into propositional logic problems connected in a deliberately counterintuitive way, isolating logic from pretraining shortcuts.
A debiasing metric. We propose PartialCircular alongside Accuracy and Circular to mitigate the bias and randomness inherent in LLMs, giving a cleaner ranking of models.

Benchmark Construction

Each instance starts from a propositional logic expression verifiable by an external logic solver, is instantiated with diverse natural sentences, and is then concatenated with connectives in a counterintuitive manner. The benchmark contains three GMAT-style question types:

3c1e

The content fails to imply three of the options while implying the remaining one.

3e1c

The content implies three options while failing to imply the remaining one.

Missing Premise

The necessary proposition guaranteeing the conclusion is removed; it becomes the correct option.

Illustration of the construction process.

Evaluation

We evaluate open- and closed-source LLMs (Mixtral, LLaMA 3.3, Qwen 2.5, Gemini, GPT-3.5, GPT-4, GPT-4o, o1-preview) in zero-shot and few-shot settings. Three metrics are reported: Accuracy, Circular, and the proposed PartialCircular, which gives a clearer separation between models than accuracy alone.

Zero-shot performance across the three question types (overall, 3e1c, 3c1e, Missing Premise).

Does It Really Test Logic?

We prompt GPT-3.5 to answer without using logical reasoning (NoLR). On DivLogicEval, performance drops, but on ReClor and LogiQA2 it paradoxically rises — evidence that DivLogicEval depends on logic far more than existing linguistically diverse benchmarks.

Benchmark	Metric	Origin	NoLR	Δ
DivLogicEval	ACC	32.2	28.6	−3.6
	CIR	16.4	15.8	−0.6
	PC	6.3	5.2	−1.1
ReClor	ACC	57.3	59.8	+2.5
	CIR	32.1	35.0	+2.9
	PC	45.1	48.0	+2.9
LogiQA2	ACC	51.9	53.2	+1.3
	CIR	15.9	20.7	+4.8
	PC	28.0	33.6	+5.6

Δ = Origin − NoLR. Negative on DivLogicEval (logic helps); positive on ReClor/LogiQA2 (other skills dominate).

BibTeX

@article{chung2024divlogiceval,
  title     = {DivLogicEval: A Framework for Benchmarking Logical Reasoning Evaluation in Large Language Models},
  author    = {Chung, Tsz Ting and Liu, Lemao and Yu, Mo and Yeung, Dit-Yan},
  journal   = {arXiv preprint arXiv:2509.15587},
  year      = {2025}
}