Duke study finds chatbots mirror some human legal judgments but diverge
Researchers compared 500 people’s responses with 26 language models across 25 legal scenarios; model answers were more uniform and sometimes more favorable to institutions.
Event / data period:
Published: · Updated:
Duke Law summarized the research in a 15 October report. The study compared answers from 500 people and 26 AI models—including systems from ChatGPT, Claude, Gemini, Llama, Grok and DeepSeek—across 25 legally relevant scenarios asking what would be “reasonable.” The paper had appeared as an arXiv preprint and was accepted to the AAAI/ACM Conference on AI, Ethics, and Society.
Researchers reported that model answers broadly tracked human judgments but clustered more narrowly than people’s responses. In some scenarios, models gave government and corporations more leeway than individuals; their answers also tended to align more with older, white, male and college-educated respondents. The team says the findings need further testing with more diverse participants and a broader range of chatbots.
Data
Legal-reasonableness study
Assessments of what was “reasonable” across 25 legal scenarios.
- · Occurred
Human and model comparison
500 participants, 26 AI models and 25 scenarios.
Methodology
Methods and findings as reported by Duke and the arXiv paper. Results are limited to the study sample and scenarios; they do not establish validated courtroom decision performance.
Updated:
Subjects and places
Subject proposals
- Science and technologySuggested
- Preserved editorial category/topic; broad candidate onlylaw.duke.edu ↗
Geographic references
Only source-provided geographic references are shown. Opening an external map is optional. Optional story maps use OpenStreetMap. Opening a map sends your IP address and this site’s address to the map provider; location data remains readable as text without loading the map.
- United States · Place role: Coverage · Place type: Country or area · Location precision: RegionalEvidence
- Explicit preserved editorial country/region; no city or national centroid inferredlaw.duke.edu ↗