Estrangement Machines

Every legal scholar has learned the difference between a court interpreting a statute and a court making new law – between applying the legislature’s policy choice and surreptitiously replacing it with one’s own. Doctrines of the separation of powers and judicial legitimacy rest on this distinction; where the line falls has been debated for as long as there have been courts.

Now suppose you give a machine a few thousand judgments and ask it to sort the passages by similarity. Will it find that line? More likely, it will not. A court developing the law tends to write much like a court simply applying it, and it may have good reason to: to name one, innovation presented as ordinary interpretation entails less reactance. The machine will identify something – cases about divorce, public procurement, employment discrimination – but we are not sure the boundary on which judicial legitimacy rests would be among its groupings.

Suppose that is right. It would be easy to conclude this is another thing AI cannot do yet. A more interesting reading is that the failure tells us something about the distinction rather than about the machine: perhaps the line between interpretation and lawmaking is not marked in the text of judgments at all, but drawn retrospectively by legal scholars, legislators, and other courts.

Why does this matter? Transcription, translation, and coding are already being delegated to AI models in empirical research, usually with little discussion of what that does to the interpretive work involved; what remains open is whether anything is gained beyond doing the same work faster. The question becomes even more acute in a collaboration like ours, which brings together methodologically distinct – sometimes conflicting – traditions.

Qualitative research, and ethnography in particular, is context-dependent and built on a handful of carefully selected cases analyzed in depth. Its guiding posture involves immersion in the field while approaching it with an open mind and, so far as possible, suspending ethnocentric assumptions. It therefore entails a degree of estrangement from the familiar, together with sustained attention to what the particularities of each case reveal about the broader context.

By contrast, quantitative and computational methods are designed to abstract away from the singularity of cases. Their guiding posture is that regularities become visible only in aggregate: the individual case is an instance of a general pattern, and analytic power increases with the number of observations. Where ethnography seeks depth through immersion, computation seeks coverage through comparison; its virtues are replicability and scale rather than closeness. The two also relate differently to assumptions. The ethnographer suspends her preconceptions deliberately, as an act of method; a model has none to suspend, but it brings those inherited from its training data and parameters. That difference is what makes the pairing uncomfortable, and what makes it productive.

Our claim, proffered as an argument rather than a finding,¹⁾ is that computational methods may facilitate a form of estrangement valuable to qualitative and, in particular, ethnographic inquiry: they can help make the familiar strange. Machines may be most useful precisely when they fail to reproduce our categories; they may be less informative when they appear simply to confirm them. Despite their prima facie opposite inclinations, the two approaches can therefore produce considerable synergies.

Our object throughout is the written record of a practice rather than the practice itself: a judgment, a hearing transcript, an interview transcript, a set of fieldnotes. Each is an account produced by someone with reasons to present things in a particular way, and each captures only the part of a practice that has been written down and can therefore be read by a machine. This reframes the usual question of whether AI can “accurately” assist qualitative research: a model that reproduces our codes perfectly merely confirms categories we have already imposed. It is the mismatch that gives us something to investigate.

What the machine is doing

Discussions of AI in research often stay at the level of “using AI.” It helps to be concrete, because the stakes depend on the task.

The first is transcription, usually treated as merely technical, whether of an interview or a court hearing. But transcription errors are not spread evenly: they cluster around accented speech, dialect, people talking over one another, and specialist vocabulary, so the transcript is least accurate where speech departs most from the standard the system assumes – the witness testifying in a second language, for instance. Translation has a similar blind spot: a model supplies the nearest equivalent of a word rather than flagging that there is none, though the missing equivalent may be what interests the researcher most.

The second is annotation, where the researcher defines the categories and the model assigns labels. This can work surprisingly well: in one careful evaluation, a general-purpose model outperformed crowd workers on several annotation tasks and agreed with itself more consistently than trained annotators.²⁾ But consistency is not accuracy: a model can be perfectly reliable and reliably wrong.³⁾ When two human coders disagree, their discussion of what a category means often exposes a conceptual problem; a model does not have that conversation. A hand-coded sample therefore remains essential, and the categories on which the model performs worst deserve the most attention: they might mark concepts (e.g. allocative justice) we struggle to fill with concrete content (e.g. first come, first served).

The third is uncovering structure that nobody specified, often described as letting the data speak. The model represents each passage as a long list of numbers, and passages whose numbers sit close together are grouped. But “close” means similar in language, not necessarily in ideas: two passages may be grouped because they share a register while saying opposite things. Nor is the grouping automatic – how many groups to identify, and what counts as a passage, shape it.

Three features characterize all three tasks. Instability: answers shift with how a question is phrased and with the order in which options are listed, and models are regularly updated and retired, so the same query may produce different results six months later.⁴⁾ Saving raw outputs and recording model versions is therefore essential. Agreeableness: a model has no stake in challenging your hypothesis, so asked whether a passage expresses distrust of the courts, it may find distrust wherever prompted to look; an explicit “none of these” option, and reruns with different prompts, help reveal how much of what it finds was put there by the question. Reduced cost: clustering ten thousand documents now costs little more than two hundred. Qualitative research has long relied on purposive sampling for two main reasons: the conviction that in-depth analysis of “information-rich cases”⁵⁾ reveals much about the broader context, and the practical constraints imposed by the time-intensive nature of sustained engagement with the field. The second reason has largely fallen away, and comprehensiveness now looks like an improvement. But the first was never a concession to scarcity, and it now has to be argued for rather than assumed.

Data governance deserves fuller treatment than we can give it here, but the relevant question is not whether AI was used. It is where the data go: a commercial interface, a provider under contract, and a model run on university hardware are not interchangeable uses of “AI.”⁶⁾

Another source of bias, or a way of exposing our own?

A common reaction is to treat AI as one more threat to objectivity, and an extensive literature shows that it does carry over human biases: language models asked to decide hypothetical criminal cases were more likely to convict, and to sentence to death, defendants whose statements were written in African American English rather than Standard American English.⁷⁾ But this frames the issue in terms foreign to qualitative research, which does not regard eliminating bias as the goal: the researcher’s positionality and interpretive choices are understood as integral to the knowledge-production process. What AI might offer is not objectivity but visibility — classifications that do not follow the categories we brought to the material.

Legal scholars are especially vulnerable to surreptitious positionality. Besides being natives of the system we study, most of us are trained neither to reflect on our positionality nor to articulate how it shapes our research. We reason in categories so habitual that we no longer notice them as artefacts — historically contingent ways of carving up a field that could have been carved otherwise. The problem is not ours alone: anthropologists studying their own societies (“at home”⁸⁾) face a version of it, as does anyone studying an institution they were trained in. Estranging such environments is hard for the reason that matters: fluency gets in the way. A generalist language model, without legal training, is at least less constrained by it.

Before leaning too heavily on this, we need to confront the obvious objection: computational methods are in practice deeply conservative. They recover genre, register, and topic — what is frequent and strongly signaled in the words themselves, usually what everyone already knows about the corpus. Disruption is the occasional and unpredictable outcome, not the usual one. Estrangement cannot be the routine product of a clustering algorithm; it has to be looked for somewhere more specific.

Against this backdrop, the value of the machine is not that it proposes better categories or is unbiased, but that it may fail to reproduce ours. This brings us back to interpretation and lawmaking. If a distinction we treat as fundamental leaves no trace in the machine’s clustering, and the method could have detected it, that is informative: not that the distinction is unreal, but that it is not carried by the language of the record. The condition matters: before reading anything into a machine’s silence, we should check that the same procedure recovers a distinction the text is known to carry. A category can be doctrinally central but linguistically invisible, and that invisibility may be less accidental where courts have an interest in not being seen to cross a line.

The converse would be equally useful. Consider the “best interests of the child,” a guiding principle in family law. It is plausible that a sorting of judgments invoking it would cluster differently depending on whether the case concerns adoption, contact, or removal — not because the doctrine is incoherent, but because one label may describe several distinct practices.

One of us has observed a similar phenomenon across entire legal systems. Clustering economies by World Bank indicators of both formal law (“on the books”) and its implementation (“on the ground”) does not reproduce the familiar legal families: common-law economies fall into markedly different clusters, separated less by inherited templates than by the gap between the books and the ground.⁹⁾

These are the same test run in opposite directions: in the first we look for a category we expect but the material does not support; in the second, for unexpected categories into which it divides. Neither is specific to case law — a code applied confidently across forty transcripts might equally fail to correspond to anything in how respondents speak. In practice this means withholding the codebook, letting the machine sort the corpus without it, and comparing its groupings with the human-generated codes — not to establish which is correct, but to treat the mismatch as an object of inquiry. We call this “mismatch analysis.”

The objection from ethnography

Much of this will strike ethnographers as beside the point, and their concerns deserve consideration, not least because they highlight two genuine limitations.

First, the corpus is not the field. Ethnographic knowledge comes from being present over time: from observing what people do, which may differ from what they say they do; from exchanges that continue after an interview has “officially” ended; from the pause before an answer; from noticing who stays silent, and who is never invited to speak. A transcript records some of this, but it is one account of an encounter whose meaning unfolds in a wider context. Nor does the text of a judgment capture what it means to the person it is addressed to, let alone what they do with it afterward.

This is where the two halves of our subject converge. A judgment is a court’s account of its reasons, not a record of how the decision was reached: curated, retrospective, sometimes a rationalization of a decision arrived at on other grounds.¹⁰⁾ An interview transcript is likewise mediated, produced with someone who knows they are being recorded. What the fieldworker observes and the microphone misses has its equivalent in judicial chambers — the deliberation, the exchanges between judges and clerks, the abandoned draft. In both settings the most polished and least revealing layer is the one available at scale.¹¹⁾

We accept this as a limit on what computational methods can claim, and it should discipline how findings are described. “The court’s reasoning changed” and “the language in which the court justifies itself changed” are different claims, and only the second is supported by a computational analysis of judgments.

Second, fieldnotes are already interpretation. This is the more fundamental objection, and the one least often addressed. Fieldnotes are written in the field or shortly after and often revisited later, and at each stage the researcher chooses what to record, how to describe it, and what to leave out.¹²⁾ What a model receives is that written rendering, not the encounter. It is not analyzing the field; it is analyzing a researcher’s account of it, produced with particular attention and particular blind spots.

Here, however, the objection points toward using computational methods rather than away from them. If a model is coding the researcher’s interpretive residue, then disagreement between machine and human coding is not a measurement of the field at all. It is information about the researcher: what they consistently noticed, described in similar terms, and treated as belonging together. Those habits of attention are exactly the reflexive object anthropology has spent decades trying to make visible, and they are unusually hard to see from within. The machine would be not a substitute analyst but a mirror held at an unfamiliar angle.

We do not fully agree about how far this goes. One of us thinks the mirror argument answers the objection: a researcher confronted with a strange partition of their own notes may learn something no amount of positionality writing would surface. The other thinks it merely shifts the problem, since what the mirror reflects is that attention filtered through a model with commitments of its own, and the two cannot obviously be disentangled. We leave the disagreement standing because what one can conclude from a machine’s output is precisely what is contested.

The problem of the outlier

In quantitative research, the unusual case is normally treated as noise, and a good statistical method is praised for ignoring it. In qualitative research, by contrast, it may be exactly what we want to understand. Doctrinal legal scholars reach first for the landmark judgment, unusual almost by definition: it departs from earlier reasoning, is hard to predict, and matters because it marks a change. A model that predicts ordinary cases well may be least useful when the law is moving.¹³⁾ But the cases that interest qualitative researchers, and anthropologists in particular, are often the opposite: an unreported decision, a dispute resolved outside the formal court system, a claim abandoned before it reached anyone’s attention. What these share with the landmark is not prominence but fit — each fits the pattern badly, and each is where the researcher learns the pattern has an edge.

Instead of asking what a model gets right, we might ask what it gets wrong, and treat the outlier as a reason to look more closely rather than a problem to remove. This can be done simply. Train a model to predict something about each document and look at the cases it gets most badly wrong; ask a language model how surprising it finds each document, and read the strangest; or use a standard anomaly detector, which identifies passages sitting far from everything else.¹⁴⁾ These tools are mature and normally used for cleaning data; repurposed, they offer a way of deciding where to look — and of asking why certain cases reveal a category the annotation scheme has missed or a change in institutional practice.

Two cautions. Most unusual cases are unusual for uninteresting reasons: an exceptional factual setting, a transcription error, a procedural peculiarity. The machine can tell us something is different; only close reading tells us whether it matters. And an outlier is always an outlier relative to something, so it may tell us as much about the archive as about the phenomenon.

What follows

None of this argues against using AI in qualitative research, only against the version in which it replaces interpretation. The more productive possibility is iterative rather than substitutive — close to what has been called computational grounded theory, in which computational analysis raises questions that researchers then investigate through close reading.¹⁵⁾ “Mismatch analysis” is one version of this approach: withhold the codebook, compare the machine’s groupings with ours, read the disagreements in both directions, and look closely at the cases the model finds hardest to place. The point is not for the machine to have the last word, but for its output to send us back to the material.

So, can we compute the human factor in empirical legal research? A qualified yes — over a narrower domain than its advocates claim, and for a different purpose: not to settle questions faster, but to test which of our categories the record actually carries and which we have been supplying ourselves. The more useful question may not be whether AI can become an objective researcher, but whether it can become a useful, even partly biased interlocutor — one that unsettles categories we take for granted and points us toward cases we would otherwise overlook.

The machine will not tell us where interpretation ends and lawmaking begins. It may, however, show us how much of that line we have drawn ourselves.