When AI Meets Finance (stockagent): A Benchmark for Simulating Large Language Model Behaviors in Controlled Trading Environments

Can AI Agents be benchmarked within strictly controlled simulated trading environments to investigate how external factors impact their collective trading behaviors? These factors, which frequently influence trading behavior, are critical elements in the quest to maximize investors’ profits. Our work aims to address this problem by utilizing large language model-based agents. We have developed a multi-agent AI system, StockAgent, driven by LLMs and designed to systematically explore simulated trading behaviors in controlled environments. StockAgent enables examination of how external factors might affect agent behavior and profitability in simulations, without empirical validation for real-world use. Additionally, StockAgent avoids the test-set leakage issue present in existing AI-agent-based trading simulation systems. Specifically, it prevents the model from leveraging prior knowledge it may have acquired related to the test data. We evaluate different LLMs within the StockAgent framework, which serves as a benchmark for testing LLM behavioral tendencies with rigorous internal validity checks and non-LLM baselines. The experimental results demonstrate the impact of key external factors on stock market trading, including trading behavior and the rules governing stock price fluctuations. This research examines the phenomenon of agents’ free-trading gaps in the context of no prior knowledge of market data. The patterns identified through StockAgent simulations offer methodological insights into LLM behaviors in simulated financial environments. The code is available at: https://github.com/MingyuJ666/Stockagent .