The controversy surrounding COMPAS exposes a significant gap between computer science and social science in understanding bias, highlighting the need to align computational fairness metrics with humanistic interpretations. In response, we introduce the EVENS benchmark to align the Equality versus Equity Notion Spectrum in LLMs. Our contributions include constructing an equality-equity notion spectrum and generating a corresponding dataset of key fairness scenarios, evaluating models' initial stances and test stance adjustments under external legal regulations and internal organizational regulations using Retrieval-Augmented Generation (RAG), introducing Chain-of-Thought (CoT) prompting to guide fairness reasoning, and adding an uncertain choice to assess its impact. Our findings indicate that LLMs initially favor equality over equity. Incorporating legal and organizational regulations of equity through RAG can reduce proportional equality in most models and enhance equity recognition in GPT4o significantly. CoT improves the equity reasoning of Chinese models but may also rationalize existing biases, and the uncertain option promotes more cautious responses. The links to the code and datasets: https://github.com/CrexCheng/EVEN.
The legislative support system developed in this study is a subsystem of the eLen ordinance database, which provides public access to ordinance data covering approximately 97% of all municipalities in Japan. The database has been used by about half of the municipalities since 2012, and its subsystem provides legislators with the ability to automatically synthesize ordinance templates loaded in 2022. The basic mechanism of this subsystem is to cluster ordinances that have many similar expressions, extract their shared expressions to create templates, and create a selective list of differences. Since the clustering must focus on textual expressions, the system constructs a graph of similar ordinances using similarity information based on an edit-distance (the Levenshtein distance) of over 100 trillion article combinations, and clusters adjacent ordinances.
The use of machine and deep learning techniques to predict outcomes in legal proceedings is a highly debated topic among legal scholars and policymakers. These technologies have the potential for supporting judicial decision-making, assist litigants, and analyze biases within the legal process. However, challenges remain, notably in the reluctance of judges to adopt such tools due to concerns over judicial independence, the normative correctness, accuracy, and robustness of algorithmic decisions, and the transparency of AI systems. It is claimed that methods are needed to validate AI-based judicial predication mechanisms. This paper contributes to addressing these challenges by developing a rich computational normative framework for judicial case-based reasoning grounded in Defeasible Deontic Logic. We explore legal CBR, focusing on inconsistencies and incomplete knowledge within case bases, and emphasize the importance of normative explanations to ensure transparency and justification in legal decision-making. By reconstructing CBR and deontic explanations, we provide a formal mechanism for validating AI-based judicial predictions where cases are represented using a fine-grained deontic language.
Unfair clause detection is an extremely useful AI application for consumer protection. Artificial intelligence has recently been successful in building systems capable to automatically detect unfair clauses in Terms of Service, and also to identify their unfairness categories. Since Large Language Models (LLMs) are nowadays bringing a revolution to the field of artificial intelligence, and in particular to natural language processing and understanding, in this paper we compare several different prompt strategies for LLMs with more traditional BERT-based fine-tuned models. Our extensive experimental evaluation aims to investigate whether it is worth using LLMs also for this challenging domain-specific task.
Computational models of legal reasoning often employ factors to reason about cases. Factors can be used to analogize a current factual scenario and precedents and to make arguments for or against a conclusion. Courts not only determine whether a factor applies to a case or not, but often how strongly the factor applies, that is, the factor's magnitude in the case. Previous methods for automatically extracting factors from cases cannot identify factors' magnitudes. We present and evaluate a method employing Large Language Models (LLMs) to identify factor magnitudes using fewshot prompts with or without Wordnet definitions. We also show how the extracted magnitudes can be used in constructing legal arguments that employ factors and magnitudes the way judges and lawyers do.
Transformer-based models, such as BERT, have proven effective in capturing textual patterns for classification tasks, yet legal texts present unique challenges. These challenges arise from subtle distinctions between document types and the absence of clear syntactic signals or keywords, requiring large, high-quality datasets and sophisticated techniques to capture these nuances. Large language models (LLMs) offer a compelling alternative, particularly when combined with in-context learning strategies that reduce reliance on extensive labeled datasets. However, the effectiveness of LLMs in such tasks is heavily dependent on the design of effective prompts, which remains a complex and nontrivial task for non-AI specialists. To address these challenges, we implemented, within the DSPy framework, PromptBreeder, a genetic algorithm-based optimization technique to automate the creation and refinement of prompts. Starting from generic and legal-oriented seed sentences, our approach consistently generated prompts that outperformed both BERT models and LLM classifiers relying on domain expert-designed prompts.
Autonomous vehicles (AVs) must comply with regulatory frameworks to ensure road safety and predictability. Current proposed AV systems predominantly rely on machine learning models that lack explicit, computable representations of traffic laws, raising concerns about accountability and robustness in complex scenarios. This study proposes a novel pipeline that embeds formal logic rules in the autonomous agent to ensure legal compliance. To address the knowledge acquisition bottleneck, we propose using Large Language Models (LLMs), robust prompt engineering, and Logical English (LE) to translate traffic rules from natural language into a human-readable, executable rule-based framework. The pipeline includes an error correction phase to refine the process of extracting legal rules, which are then integrated into a simulation environment. Our approach successfully performed the translation of legal text into a structured, computable format, improving the transparency and interpretability of the high level decision making. The error correction phase improves rule accuracy, while simulations further validate rule compliance and performance in dynamic traffic scenarios.
Tax foreclosure proceedings present a significant challenge to judicial systems worldwide, with Brazil alone managing 24 million active cases requiring an average of 6.8 years for resolution. This paper presents OSIRIS, a novel framework that combines pattern recognition, embedding-based classification, and finite state modeling to automate and optimize tax foreclosure case processing. Through analysis of 250 lawsuits, we mapped 273 distinct legal acts, 44 states, and 10 phases, creating a comprehensive model of tax foreclosure workflows. Our framework employs a three-step classification approach that progressively applies more sophisticated techniques as needed, achieving 88% accuracy in act classification while maintaining computational efficiency. The system outperforms standalone large language models in state prediction (82% vs 27% accuracy) while reducing token usage by 96%. Implementation in the Federal District Court demonstrates OSIRIS's potential to significantly reduce the current USD 4,900 per-case processing cost and accelerate case resolution. This work contributes to the growing field of legal automation by providing a scalable, transparent approach that enhances judicial efficiency while preserving procedural integrity and human oversight.
In legal AI, prompt engineering unlocks legal knowledge by bridging experts and LLMs. These prompts, often not following human language conventions, boost LLMs' NLP performance with minimal data, yet task-specific optimization remains challenging. Current soft-prompt methods lack interpretability and cross-linguistic adaptability, while general optimization approaches prove time-consuming and fail to integrate legal reasoning. This paper proposes LAC-APO, a multi-agent collaborative framework based on Toulmin's model that incorporates judges' implicit knowledge. Through Logical combing-Knowledge injection-Error induction-Prompt adjustment process, it systematically converts implicit expertise into explicit LLM outputs. Experiments show LAC-APO outperforms manual optimization and existing competitiveness prompt optimization methods, enhancing efficiency while maintaining legal reasoning integrity.
This paper introduces ALEXChat, a hybrid generative-symbolic chatbot designed to enhance the accessibility and explainability of legal expert systems. It integrates symbolic reasoning with large language models (LLMs) to bridge the gap between complex legal reasoning and user comprehension. ALEXChat combines the extraction of legally relevant facts from scenarios, rule-based prolog reasoning and an argumentation framework for conformity assessment, with the linguistic power of LLMs for accessible user fruition.
AF-CBA uses case-based argumentation to justify classifier predictions by arguing about differences between cases. We extend the mechanism by modelling which differences can compensate for each other by constructing arguments using domain knowledge. This involves a secondary argumentation framework. To assist experts in defining the appropriate domain knowledge, we use a rule-based classifier for semi-automated knowledge induction. We use the resulting rule set to derive arguments and demonstrate this with an evaluation procedure.
Reporting requests in EU legislation require institutions to submit reports for legislative monitoring and policy implementation. As these meta-norms grow in complexity, compliance becomes increasingly challenging and burdensome. This paper leverages a Hybrid AI approach to detect, extract, and track reporting requests within the EU legislation using the RRVM ontology and European legal standards (AKN4EU, CELLAR, ELI). We investigate four interconnected and integrated areas: (1) detecting reporting requests and their concepts in legislative texts, (2) navigating normative references, (3) converting extracted legal knowledge into RDF for a Knowledge Graph, and (4) tracking modifications of reporting requests over time. Our approach compares machine learning (ML) and large language models (LLMs) for detection, demonstrating the advantages and limitations of both. By structuring Reporting Requests into a dynamic knowledge graph, our method improves the handling of Reporting Requests, reduces their administrative burden, and supports better legislative drafting and policy monitoring.
Normative diagrams are a legal visualisation method with promising applications within legal practice, legal theory, and especially for the didactics of law. The paper at hand presents the implementation of ViNo, a legal visualisation tool (including a graphical user interface) based on this method.
We report on a first set of results from experiments undertaken to tackle the novel task of determining the optimal context window for extracting and contextualising citation instances within case law. The wider task of outcome prediction using AI tools cannot be undertaken without considering the role that citations play when new cases are being decided. This short paper aims to shine a light on the importance of this task and provide the foundation for developing AI tools to capture citations' context by examining a corpus of legal cases taken from the European Court of Human Rights. Our results show that there is an identifiable "Goldilocks Zone" of scoped paragraph-level context windows that attention can be focused on for extracting citation instances.
This study introduces an approach for evaluating legal Large Language Models (LLMs) through lawyer preference ratings rather than traditional benchmarks. Using a pregnancy discrimination case, we compared the outputs from foundation models, enterprise legal tools, and human lawyers across four litigation tasks. Senior partners conducted blind reviews of the outputs. The results provided empirical color to long-held assumptions: enterprise tools demonstrated remarkable consistency, while both LLMs and human lawyers showed striking performance swings. More importantly, the study reveals that traditional metrics may be fundamentally misaligned with what experienced lawyers actually value in legal work. The findings challenge existing benchmarking approaches and suggest that comprehensiveness may be less important than clarity and precision in legal work products.
In the legal field, accurately interpreting and applying legal definitions is crucial yet challenging due to inherent ambiguities. This paper introduces DeCoRA, a novel framework that enhances legal argumentation by incorporating context-based reasoning to address these ambiguities, with a focus on the judge as the central decision maker. Unlike black-box models, such as generative models or outcome prediction systems, which often produce outputs without fully explaining the reasoning behind their conclusions, DeCoRA emphasizes transparency by modeling the judicial decision-making process in a structured and interpretable manner. Our key contributions include: (1) a tree-based knowledge base that organizes legal definitions, highlighting their relationships and effects; (2) a context-aware definition framework enabling judges to interpret definitions considering legal and contextual relevance; and (3) an effective method for handling complex legal scenarios with conflicting or overlapping definitions.
We proposed an innovative solution through an Artificial Intelligence driven legal analysis customized to the utility of the Nepalese legal context. Using advanced machine learning (ML) models and Retrieval-Augmented Generation (RAG) techniques, the research provides legal insights, streamlines judicial processes, and enhances accessibility to legal information. The legal documents were processed to convert into JSON format, and then to convert into vector data. GPT-4o was used for query expansion and response generation, whereas text embeddings were generated through text-embedding-ada-002. Key features include efficient document retrieval and query expansion for enhanced search precision. The model performs well across different query types, achieving an F-1 score of 0.797 for rule-recall, 0.857 for rhetorical understanding, and 0.875 for interpretation-based queries. This work marks a significant step towards integrating AI into the legal domain of Nepal.
Tunnel vision and confirmation bias can lead to miscarriages of justice. A way to avoid tunnel vision is to consider your evidence in light of more than one scenario. Alternative scenarios allow us to consider how probable each scenario is, compared to the other considered scenarios. Bayesian Networks have been proposed as a formal method for reasoning about the probability of scenarios. Specifically, alternative scenarios were modelled using Bayesian networks with a constraint node, which ensures mutual exclusivity. However, the performance of these methods in situations where not all possible alternative scenarios are modeled, the non-exhaustive setting, has not been investigated. Since it is impossible to explicitly cover everything that could possibly have happened in a model, it is important to know how these methods handle non-exhaustiveness. We evaluate four methods using an agent-based model that simulates an environment in which a crime could occur. Taking this as the ground truth, we compare different Bayesian network modeling methods on five aspects related to the quality of the representation of the ground truth as well as computational performance. We find that some methods result in disparities between the ground truth and the predicted posterior probabilities for the scenarios in a non-exhaustive setting. In an exhaustive setting, the proposed methods perform well. The construction approach that models scenarios in terms of conjunctions of events performs well in both settings.
Class imbalance poses a significant challenge in many legal text classification tasks that has remained mostly unaddressed. In the field of computer vision, in contrast, various loss functions have been developed to tackle the issue. Our study evaluates the proposed loss functions as possible drop-in solutions for transformer models pre-trained on legal data. We conduct extensive experiments that include the training of over 860 LexLM RoBERTa-large and PoL-BERT-large models for up to 20 epochs on the popular ECHR and Unfair-ToS datasets. Our results highlight that via minimal modifications to existing fine-tuning pipelines, custom loss functions can improve model performance and training behavior. We also show that while standard binary cross entropy paired with weighted random sampling leads to an increase in training time, it can be an effective tool that does not need hyperparameter tuning.
