Adjudicating traffic-crash liability is a critical legal task, yet it remains time-consuming and dependent on expert interpretation of unstructured narratives. General-purpose large language models (LLMs) can produce unsupported or difficult-to-audit outputs when liability assessment requires structured, multi-step legal reasoning. We therefore propose Law-CoT, a two-stage framework that generates evidence-grounded and auditable liability assessments from narrative descriptions. Grounded in civil-law liability analysis, Law-CoT contains three core properties: (i) Sequential Reasoning, implemented through programmatic gating to enforce the party-specific fault-to-causation dependency, while case-level damage existence and statutory sanction triggers are assessed separately; (ii) Evidence Grounding, which requires conclusions to be supported by verbatim spans from the narratives; and (iii) a Closed-World Constraint, which restricts legal reasoning to a retrieved set of statutes. On 800 traffic-crash reports, Law-CoT achieves a Micro-F1 score of 91.52
The legal field already uses various large language models (LLMs) in real-world applications. The practical use in the patent domain concentrates on general pre-trained and foundation models without domain specificity. However, little is known and documented about their performance; their limitations are poorly studied and documented. Quality control and any targeted improvement of natural language processing in patent law require such information. We evaluated several open-source and proprietary LLMs—including the GPT series, Anthropic, DeepSeek, and Llama-3 variants—on parts of the European qualifying examination (EQE) for future European patent attorneys. The chosen tests are intentionally technical and designed with little room for interpretation so that models cannot hide wrong associations and answers behind apparent eloquence. OpenAI o1 led with 0.82 accuracy and 0.81 F1 score, whereas AWS (Amazon Web Services) Llama 3.1 8B lagged at 0.50 accuracy, and a Python-deployed Llama 3.1 8B scored 0.55. The latter two are within the range of mere guessing for the two-option forced-choice design. None of the evaluated models could have fully passed the examination, as accuracy never exceeded the average threshold of 0.90 required for professional-level standards—also not models that are regularly promoted for their assumed beyond-PhD- and bar-admitted-lawyer-level performance. GPT-4o excelled at integrating text and graphics, whereas Claude 3 Opus often struggled to maintain formatting coherence. Human patent experts evaluated the textual justifications to identify and understand problems at the argument level. They uncovered various critical shortcomings of each model. They valued clarity and legal rationale over the raw correctness of the answers, which revealed misalignment between automatic metrics and expert judgment. Model outputs were sensitive to modest temperature changes and prompt wording, which underscores the continued necessity of expert oversight. Although the arguments and our analysis of the training corpora demonstrate that the relevant legal texts were part of pre-training, the models, e.g., their embeddings, demonstrated potential for improvement in domain-specific language aspects. Future work should target logical consistency, robust multimodality, and adaptive prompting to approach human-level patent proficiency. In summary, despite the outstanding performance of recent large models, the general public might overestimate their capabilities. The field has a long way to go to develop a virtual patent attorney. This paper aims to point out several specific limitations that need solutions.
Recently, there have been opportunities and challenges for both legal practitioners and AI researchers due to the abundance of legal documents that are available in digital form. This development requires a lot of assistance and work to present this content in a helpful and distributed manner, which could have the potential to offer legal professionals many advantages. While Pretrained Language Models (PLMs) have demonstrated impressive performance in various Natural Language Processing (NLP) tasks, developing effective strategies for legal practice tasks remains challenging. This is mainly because legal documents often consist of lengthy and non-uniformly structured text, which sets them apart from general documents. Consequently, the PLMs may not fully capture the intricate legal details present in legal documents. This is because most language models struggle to grasp the extensive connections between various structures in the text. To address these challenges, we propose an approach that utilizes outputs from multiple intermediate layers of pre-trained transformer models while considering the entire legal document. Our primary objective is to generate more accurate and meaningful embedding representations, enabling the identification of semantic and syntactic relationships among sentences in the document. This approach also aims at preserving the contextual interactions with adjacent sentences, thereby enhancing the overall understanding of the document’s content. The performance of our proposed approach is showcased through experimental results conducted on publicly available legal datasets over different tasks.
Legal judgment documents contain complex legal terminology, long-range contextual dependencies, and multiple interrelated entity references, making automated legal document understanding a challenging Natural Language Processing (NLP) task. Existing studies primarily address Named Entity Recognition (NER), Coreference Resolution (CR), and Judicial Outcome Classification (JOC) independently, resulting in fragmented contextual understanding and limited semantic interaction among complementary legal NLP tasks. To address this limitation, this study proposes the Legal Unified Modular Pipeline (LUMP), a sequential transformer-based framework that integrates XLNet for extracting domain-specific legal entities, DistilBERT for resolving pronoun–antecedent relationships to maintain entity consistency, and LegalBERT for classifying judicial outcomes using contextually enriched legal representations. Owing to the absence of a unified benchmark dataset, each module was independently optimized using the most appropriate benchmark dataset and subsequently integrated during inference through sequential contextual enrichment. Experimental evaluation demonstrated that the NER, CR, and JOC modules achieved an entity-level F1-score of 93.94
Recent advances in AI make it conceivable to delegate legal decision-making to machines, or to enhance human adjudication through AI assistance. Using classic normative conflicts — the trolley problem and comparable moral dilemmas — as a proof of concept, we examine the alignment between AI legal reasoning and human judgment. In our baseline experiment, we find a pronounced mismatch between decisions made by GPT and those of human subjects. This misalignment raises substantive concerns for AI-powered legal decision-aids. We investigate whether explicit normative guidance can address this misalignment, with mixed results. GPT-3.5 is susceptible to such intervention, but frequently refuses to decide when faced with a moral dilemma. GPT-4 is outright utilitarian, and essentially ignores the instruction to decide on deontological grounds. GPT-o3-mini faithfully implements this instruction, but is unwilling to balance deontological and utilitarian concerns if instructed to do so. We replicate the experiment with four LLMs from different providers. Claude-sonnet-4.6 comes closest to human respondents. Gemini-2.5-flash-lite is most sensitive to normative instructions. Llama-4-scout and Mistral-nemo have a strong utilitarian bias, and do not strongly respond to normative interventions. At least for the time being, explicit normative instructions are not fully able to realign AI advice with the normative convictions of the population, or the legislator deciding on its behalf.
