# EXP-006: Speech Act Classification from LLM Hidden States (Austin/Searle)

**Date:** 2026-04-08
**Author:** @eazevedo
**Tags:** #probing, #speech-acts, #pragmatics, #llm-internals, #qwen, #philosophy

## Question

Can a pre-trained language model distinguish between speech act types (assertive, directive, commissive, expressive, declarative) in its hidden states?

## Setup

Model: Qwen2.5-7B (4-bit quantized) on RTX 3060 Ti (8 GB VRAM). Four analysis parts: (A) binary performative/constative probe — 30 sentence pairs, logistic regression with 5-fold CV at each of 28 layers; (B) next-token distribution divergence (JSD) between performative and constative utterances; (C) commitment tracking across 8 scenarios (promises, bets, declarations); (D) 5-way speech act classification using Searle's taxonomy — 40 samples (8 per class: assertive, directive, commissive, expressive, declarative), logistic regression on hidden states, 4-fold cross-validation at each layer.

## Results

| Part | Metric | Result |
| --- | --- | --- |
| A: Binary probe (perf. vs const.) | Accuracy | 100% at ALL layers including embeddings (CONFOUNDED) |
| B: Distribution divergence | Mean JSD | 0.25 (performative) vs 0.19 (constative) |
| C: Commitment tracking | Success rate | Declarations succeed, promises fail |
| D: 5-way classification, layer 0 | Accuracy | 67.5% |
| D: 5-way classification, layer 8 | Accuracy | 82.5% |
| D: 5-way classification, layer 13 | Accuracy | 87.5% |
| D: 5-way classification, layer 22 | Accuracy | 95.0% |
| D: 5-way classification, layer 26 | Accuracy | 95.0% |
| D: 5-way classification, layer 28 | Accuracy | 95.0% |

## Key Findings

- Part A (binary probe) is confounded: 100% accuracy at the embedding layer means it separates grammatical person ('I promise' vs 'She ran'), not speech act type. Always check embedding-layer baselines when probing.
- 95% five-way speech act classification is genuine. The 5-way task forces the probe to distinguish WITHIN the same grammatical structures (e.g., 'I promise' vs 'I declare' are both first-person but different illocutionary acts).
- Layer 22 is the critical transition: accuracy plateaus at ~85% through layers 8-20, then jumps to 95% in layers 22-28. The model constructs speech-act representations in its deep layers, not through surface features.
- Performative utterances create more divergent next-token distributions (JSD 0.25 vs 0.19 for constatives), consistent with their broader pragmatic implications — performatives open wider possibility spaces for what comes next.

## Lesson Learned

Binary probing is dangerously easy to confound. 100% accuracy at ALL layers — including raw embeddings — is a red flag, not a success. The fix: always include an embedding-layer baseline, and design tasks where surface features (person, tense) cannot separate the classes. The 5-way task worked precisely because it forced within-structure discrimination.

## Tools Used

Claude Opus 4 for experiment design and code generation. Qwen2.5-7B (4-bit) as the model under study. scikit-learn for logistic regression probes. scipy for JSD computation.

---
Source: https://terminus.ink/e/2026-04-08-speech-act-classification-from-llm-hidden-states-austinsearle