Otto GmbH & Co KG (doing business as Otto Group, stylized as otto group, formerly Otto Versand) is a German mail order company and one of the world's biggest e-commerce companies. Based in Hamburg, it operates in more than twenty countries. Otto is primarily a retail e-commerce company, and as a shareholder, it also operates in e-commerce services. The Otto group has expanded into real estate and financial services.In France, Otto claims to be the leading e-commerce business through 3 Suisses. In June 2013, Otto executed a full takeover to become its only shareholder.
Chatbots are increasingly used in e-commerce for customer service, product recommendations, and sales advice. Although they offer efficiency and personalization, they also carry risks: responses may exhibit bias, contain toxic language, or apply inappropriate personalization that can erode customer trust, harm brand reputation, and conflict with emerging regulations. Existing datasets address bias or toxicity in short, user-generated texts such as tweets or comments, but they do not capture the multi-turn, transactional nature of e-commerce dialogues or the interplay between multiple ethical dimensions. To close this gap, we present FairBotBench, a German–English benchmark dataset for evaluating chatbot behavior in e-commerce. The dataset comprises 600 dialogues, generated from practice-driven scenarios and diverse user profiles, and enriched with ethically problematic variants and linguistic transformations. Each dialogue is annotated by humans across three ethical dimensions, namely bias, toxicity, and personalization. To demonstrate the usefulness and potential of the benchmark, we apply it to several SOTA LLMs in an LLM-as-a-Judge scenario, evaluating how well they can detect ethically problematic chatbot responses. FairBotBench provides the first multilingual, domain-specific, and ethically comprehensive benchmark for conversational AI in e-commerce, offering a robust foundation for research, evaluation, and regulatory compliance .
Large language models (LLMs) are increasingly used as automated judges for evaluating the behavior of chatbots and other AI systems. In deployment, however, their agreement with the intended human evaluation policy may degrade as user interactions and score distributions change. Existing automatic prompt optimization methods are primarily studied as one-shot procedures on static development sets and therefore provide limited guidance for maintaining LLM judges over time. We frame this problem as judge drift and introduce PromptOps, a closedloop framework for monitoring and iteratively recalibrating LLM-as-a-Judge (LLMaaJ) systems. PromptOps separates four operational decisions: when to optimize, which optimization procedure to apply, which feedback records to use, and how to update the judge prompt. We instantiate the framework with five prompt optimization methods: GEPA, SIMBA, Rubric Synthesis, and exemplar selection using either simulated annealing or a Tree-structured Parzen Estimator. Our experiments demonstrate that iterative prompt optimization can recalibrate LLM judges and recover performance after controlled shifts in the referencescore distribution. However, effective maintenance is inherently contextdependent: neither optimization algorithm choice nor update timing generalizes universally across settings. These findings establish prompt optimization for deployed LLM judges as an inherently iterative maintenance problem and demonstrate that PromptOps provides a practical abstraction for studying and implementing this process.
Large Language Models (LLMs) are increasingly employed as automated judges for evaluating chatbot responses. While proprietary models can achieve high agreement with human annotators, they remain costly and difficult to deploy in practical settings. Lightweight open-source LLM judges provide an attractive alternative but typically exhibit lower agreement with human ratings. In this paper, we investigate whether soft prompt tuning can align open-source LLM judges more closely with human annotation policy. We train soft prompts on the FairBotBench benchmark and evaluate the adapted judges across the dimensions of bias, toxicity, and personalization. Our findings demonstrate that hybrid soft-prompt tuning, which combines textual prompts with learned continuous prompt embeddings, consistently improves agreement with human annotations over the corresponding untuned baselines. For Qwen 3.5 9B, the adapted judges surpass both hosted reference judges across all three evaluation dimensions.
We present FairBot-Live, an interactive demonstrator for exploring ethical challenges of large language model (LLM)-based chatbots in e-commerce. The system enables users to simulate advisory dialogues with state-of-the-art LLMs and to evaluate the generated responses along the metrics bias, personalization, and toxicity. The evaluation is performed using the LLM-as-a-Judge approach, where one LLM assesses the outputs of another, providing both numerical scores and textual justifications.FairBot-Live provides a reproducible and transparent environment for investigating ethical evaluation methods and model behavior in realistic e-commerce interactions.
This work introduces TRON, a scalable session-based Transformer Recommender using Optimized Negative-sampling. Motivated by the scalability and performance limitations of prevailing models such as SASRec and GRU4Rec+, TRON integrates top-k negative sampling and listwise loss functions to enhance its recommendation accuracy. Evaluations on relevant large-scale e-commerce datasets show that TRON improves upon the recommendation quality of current methods while maintaining training speeds similar to SASRec. A live A/B test yielded an 18.14% increase in click-through rate over SASRec, highlighting the potential of TRON in practical settings. For further research, we provide access to our source code1 and an anonymized dataset2.
